Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare crawl endpoint

developers.cloudflare.com

Post reply on HN
Print view View original

151–160 of 194 posts

Re: Cloudflare crawl endpoint

#151

Post by mrweasel »

moebrowne wrote: post #131
mrweasel wrote: post #121

Earlier quoted context omitted.

> Most crawls are not that latency sensitive, certainly not the ai ones. They certainly behave like they are. We constantly see crawlers trying to do cache busting, for pages that hasn't change in days, if not weeks. It's hard to tell where the bots are coming from theses days, as most have taken to just lie and say that they are Chrome. I'd agree that the respecting robots.txt makes this a non-starter for the proble…

> We constantly see crawlers trying to do cache busting Do you have a source for this? Not saying you're wrong, I'd just like to know more

Not really, given that the work we do in that direction isn't exactly public. You can recreate the scenario though. Spin up a wiki of some sort, scrapers love wikis, ideally enable some form of caching, and just sit back and watch scrapers throw random shit in the URL parameters.

Re: Cloudflare crawl endpoint

#153

Post by tyingq »

RamblingCTO wrote: post #115

Doesn't work for pages protected by cloudflare in my experience. What a shame, they could've produced the problem and sold the solution.

That's too funny. If true, really looking forward to the Cloudflare response here. I'm unsure how you would spin that in a way that didn't seem self-serving.

Re: Cloudflare crawl endpoint

#154

Post by neversupervised »

RamblingCTO wrote: post #150
davidhariri wrote: post #148

Earlier quoted context omitted.

Came here to write this. I am getting much better results from Firecrawl (not affiliated with them, just a happy customer).

fuck firecrawl. they copied my idea by showing interest in my product and then copied it, used their YC money to give it all out for free. fuck nick in particular. I'm still salty over this

Tell more. Crawling is not a new idea. How did they abuse you?

Re: Cloudflare crawl endpoint

#155

Post by antonyh »

RamblingCTO wrote: post #115

Doesn't work for pages protected by cloudflare in my experience. What a shame, they could've produced the problem and sold the solution.

Wait. What?

Is this just a way to strong-arm non-cloudflarians into adopting their platform if you don't want your site crawled? It does sound like they are selling the solution to avoid their own content crawler.

Re: Cloudflare crawl endpoint

#156

Post by xeornet »

RamblingCTO wrote: post #150
davidhariri wrote: post #148

Earlier quoted context omitted.

Came here to write this. I am getting much better results from Firecrawl (not affiliated with them, just a happy customer).

fuck firecrawl. they copied my idea by showing interest in my product and then copied it, used their YC money to give it all out for free. fuck nick in particular. I'm still salty over this

"they copied my idea by showing interest in my product and then copied it". What exactly is revolutionary about Firecrawl or your product? Scraping APIs have been around for over a decade.

Re: Cloudflare crawl endpoint

#158

Post by oybng »

PeterStuer wrote: post #104
Lasang wrote: post #60

The idea of exposing a structured crawl endpoint feels like a natural evolution of robots.txt and sitemaps. If more sites provided explicit machine-readable entry points for crawlers, indexing could become a lot less wasteful. Right now crawlers spend a lot of effort rediscovering the same structure over and over. It also raises interesting questions about whether sites will eventually provide different views for hum…

The only ones benefitting from 'wastefull' crawling are the anti-bot solution vendors. Everyone else is incentivized to crawl as efficiently as possible. Makes you think, right?

I yearn for the days when a single kb get was enough. Now it's endless wastage spawning entire browsers larger than operating systems with mitigations, hacks and proxies. Requesting access directly from webmasters is only met with silence. All of my once simple, hobbyist programs are now bloated beyond belief and less reliable than ever

Re: Cloudflare crawl endpoint

#160

Post by paxys »

RamblingCTO wrote: post #115

Doesn't work for pages protected by cloudflare in my experience. What a shame, they could've produced the problem and sold the solution.

That’s what they are doing. This is a textbook protection racket.

“Buy Cloudflare bot protection, otherwise it would be a shame if your site got scraped and ddos’d.”

Who is doing the scraping and ddosing? Cloudflare.

Post reply on HN