Queue-It protected pages catch it as well and prevent crawling.
Cloudflare crawl endpoint
171–180 of 194 posts
Re: Cloudflare crawl endpoint
#172Earlier quoted context omitted.
Ask yourself, why would a scraper ddos? Why would a ddos-protection vendor ddos?
> Ask yourself, why would a scraper ddos? Don't need to ask anything i can tell you exactly - because they have no regard for anything but their own profit. Let me give you an example of this mom and pop shop known as anthropic. You see they have this thing called claudebot and at least initially it scraped iterating through IP's. Now you have these things called shared hosting servers, typically running 1000-10000 d…
So how would you avoid this specific situation as a web-crawler that tries to be well behaved? You strictly adhere to robots.txt as specified by each domain. The problem is not with any of the sites but the density (1000-10000) by which the hoster packed them. If e.g. the crawler had a 1 sec between page governor even if robots.txt had no rate specified, which to be fair is very reasonable, this packing could still lead to high server load.
Re: Cloudflare crawl endpoint
#173Earlier quoted context omitted.
That's too funny. If true, really looking forward to the Cloudflare response here. I'm unsure how you would spin that in a way that didn't seem self-serving.
It's very clearly disclosed in the linked docs already, it says that Cloudflare Bot Protection will block it same as all other bots, unless you choose to allow it as an exception. If they didn't do it that way, people would accuse them of either bypassing their own product (possibly anticompetitive) or just having a low quality one.
So it doesn't take any action to work around other bot protections? Feels like that would be on the list of features an AI company wanting to scrape would ask for.
Re: Cloudflare crawl endpoint
#174Fuck Cloudflare.
Re: Cloudflare crawl endpoint
#175Doesn't work for pages protected by cloudflare in my experience. What a shame, they could've produced the problem and sold the solution.
Came here to write this. I am getting much better results from Firecrawl (not affiliated with them, just a happy customer).
I'd love for you to kick the tires on https://grubcrawler.dev
Re: Cloudflare crawl endpoint
#176I remember reading a CF blog post about crawler separation and responsible AI bot principles where they argue every bot should have one distinct purpose. Now they're building crawling infrastructure themselves, and their own /crawl endpoint lists "training AI systems" as a use case alongside regular crawling. So not only are they in the crawling business now, they're not following the separation principle. To be fair, there's a business logic here. But it's hard not to notice the irony.
https://blog.cloudflare.com/uk-google-ai-crawler-policy/
Re: Cloudflare crawl endpoint
#177Doesn't work for pages protected by cloudflare in my experience. What a shame, they could've produced the problem and sold the solution.
Cloudflare crawl respects robots.txt. It does not attempt to bypass any anti-crawling measures. If the site doesn't want to be crawled -- whether it uses Cloudflare or not -- this product will not help you crawl it.
Some sites actually want crawlers -- e.g. sites that are selling a product, documentation, etc. That's what this product is meant for.
https://x.com/CloudflareDev/status/2031745285517455615
(Disclosure: I work for Cloudflare but not on this product.)
Re: Cloudflare crawl endpoint
#178Earlier quoted context omitted.
"they copied my idea by showing interest in my product and then copied it". What exactly is revolutionary about Firecrawl or your product? Scraping APIs have been around for over a decade.
I was the first to return markdown and use reader mode stuff to strip irrelevant stuff. Theres copying and there's talking to the founder sounding interested to have your team copy what I did in the background. One is fair game, the other is a dick head move.
I think that is a neat idea and it sucks this happened, but how long before somebody simply saw that feature and replicated it? I'm curious, had you considered a deeper moat than that?
This is especially relevant given AI is making this kind of thing easy at an industrial scale. I think we should all be looking for alternative moats.
Re: Cloudflare crawl endpoint
#179IMO the under-discussed risk here is that sites will start serving different content to verified crawlers vs real users. You're already seeing it with known search bots getting sanitized views. If your agent's context comes from a crawl the site knows is going to an AI, you have no guarantee it matches what a human sees, and that data quality problem won't surface until your agent starts acting on selectively curated information.
This could go wrong on same levels.
Re: Cloudflare crawl endpoint
#180Earlier quoted context omitted.
It's very clearly disclosed in the linked docs already, it says that Cloudflare Bot Protection will block it same as all other bots, unless you choose to allow it as an exception. If they didn't do it that way, people would accuse them of either bypassing their own product (possibly anticompetitive) or just having a low quality one.
So it doesn't take any action to work around other bot protections? Feels like that would be on the list of features an AI company wanting to scrape would ask for.
No, it does not take any action to work around other bot protections.
https://x.com/CloudflareDev/status/2031745285517455615
(Disclosure: I work for Cloudflare but not on this product.)