Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare crawl endpoint

developers.cloudflare.com

Post reply on HN
Print view View original

181–190 of 194 posts

Re: Cloudflare crawl endpoint

#181

Post by kentonv »

paxys wrote: post #160
RamblingCTO wrote: post #115

Doesn't work for pages protected by cloudflare in my experience. What a shame, they could've produced the problem and sold the solution.

That’s what they are doing. This is a textbook protection racket. “Buy Cloudflare bot protection, otherwise it would be a shame if your site got scraped and ddos’d.” Who is doing the scraping and ddosing? Cloudflare.

You can trivially block Cloudflare crawl via robots.txt. You don't need to buy Cloudflare's bot protection -- this is not a malicious bot.

https://x.com/CloudflareDev/status/2031745285517455615

(Disclosure: I work for Cloudflare but not on this product. I get pretty tired of the conspiracy theories TBH.)

Re: Cloudflare crawl endpoint

#182

Post by vimda »

laalshaitaan wrote: post #179

IMO the under-discussed risk here is that sites will start serving different content to verified crawlers vs real users. You're already seeing it with known search bots getting sanitized views. If your agent's context comes from a crawl the site knows is going to an AI, you have no guarantee it matches what a human sees, and that data quality problem won't surface until your agent starts acting on selectively curated…

This already happens in the opposite direction. See: news websites that drop their pay wall for GoogleBot

Re: Cloudflare crawl endpoint

#183

Post by m3047 »

Seems like it was just hours ago they started reaching out to my edge servers from their address space (Me: why is a reverse proxy service banging my servers when I'm not a customer? did some miscreant sign me up somehow?) and it was for Apple, privacy, mom and pie (a VPN service, dressed in noble aspirations). It never quite smelled like pie to me.

If you're doing threat hunting / risk enumeration, Cloudflare is no longer a passive service that miscreants hide behind, they now actively reach out and grab your privates.

Re: Cloudflare crawl endpoint

#185

Post by devnotes77 »

laalshaitaan wrote: post #179

IMO the under-discussed risk here is that sites will start serving different content to verified crawlers vs real users. You're already seeing it with known search bots getting sanitized views. If your agent's context comes from a crawl the site knows is going to an AI, you have no guarantee it matches what a human sees, and that data quality problem won't surface until your agent starts acting on selectively curated…

[flagged]

Re: Cloudflare crawl endpoint

#186

Post by gopher_space »

keeda wrote: post #178
RamblingCTO wrote: post #164

Earlier quoted context omitted.

I was the first to return markdown and use reader mode stuff to strip irrelevant stuff. Theres copying and there's talking to the founder sounding interested to have your team copy what I did in the background. One is fair game, the other is a dick head move.

I think that is a neat idea and it sucks this happened, but how long before somebody simply saw that feature and replicated it? I'm curious, had you considered a deeper moat than that? This is especially relevant given AI is making this kind of thing easy at an industrial scale. I think we should all be looking for alternative moats.

Sometimes timing is your moat and that's all you need. That being said I'll probably start limiting my public releases to revolve around standards I want implemented.

I'm rethinking the sources of value moats are built around. It seems like the landscape is changing and dimensions such as location, perspective, experience, and attention weigh more than they used to.

> but how long before somebody simply saw that feature and replicated it?

This is a good example. The, idk, "value store" of your org just switched from products and services to the employees who understand your process from a couple angles and can write well.

Re: Cloudflare crawl endpoint

#187

Post by sireat »

kseniamorph wrote: post #176

I remember reading a CF blog post about crawler separation and responsible AI bot principles where they argue every bot should have one distinct purpose. Now they're building crawling infrastructure themselves, and their own /crawl endpoint lists "training AI systems" as a use case alongside regular crawling. So not only are they in the crawling business now, they're not following the separation principle. To be fair…

One has to be highly suspicious of any "fair, better for others" claims coming from corporate entities.

It is the ages old story of https://en.wikipedia.org/wiki/Quod_licet_Iovi%2C_non_licet_b...

Also brings back the irony now apparent in original Google paper: http://infolab.stanford.edu/pub/papers/google.pdf "To make matters worse, some advertisers attempt to gain people’s attention by taking measures meant to mislead automated search engines."

Re: Cloudflare crawl endpoint

#188

Post by ryan14975 »

jasongill wrote: post #3

I'm surprised that Cloudflare hasn't started hosting a pre-scraped version of websites that use Cloudflare's proxy - something like https://www.example.com/cdn-cgi/cached-contents.json They already have the website content in their cache, so why not just cut out the middle man of scraping services and API's like this and publish it? Obviously there's good reasons NOT to, but I am surprised they haven't started offeri…

This makes a lot of sense. Cloudflare already has the rendered content at edge — serving a structured snapshot from cache would eliminate redundant crawling entirely.

What I'd love to see is site owners being able to opt in and control the format. Something like a /cdn-cgi/structured endpoint that respects your robots.txt directives but gives crawlers clean markdown or JSON instead of making them parse raw HTML. The site owner wins (less bot traffic), the crawler wins (structured data), and Cloudflare wins (less load on origin).

Re: Cloudflare crawl endpoint

#190

Post by Muromec »

selcuka wrote: post #39
jasongill wrote: post #3

I'm surprised that Cloudflare hasn't started hosting a pre-scraped version of websites that use Cloudflare's proxy - something like https://www.example.com/cdn-cgi/cached-contents.json They already have the website content in their cache, so why not just cut out the middle man of scraping services and API's like this and publish it? Obviously there's good reasons NOT to, but I am surprised they haven't started offeri…

Not the same thing, but they have something close (it's not on-by-default, yet) [1]: > Cloudflare's network now supports real-time content conversion at the source, for enabled zones using content negotiation headers. Now when AI systems request pages from any website that uses Cloudflare and has Markdown for Agents enabled, they can express the preference for text/markdown in the request. Our network will automatica…

so... we will get the reader moder with one header set in a browser?
Post reply on HN