Cloudflare crawl endpoint • Hacker News Board

Re: Cloudflare crawl endpoint

#61

Post by catlifeonmars » Wed, Mar 11, 2026, 1:48 AM UTC

Lasang wrote: post #60

The idea of exposing a structured crawl endpoint feels like a natural evolution of robots.txt and sitemaps. If more sites provided explicit machine-readable entry points for crawlers, indexing could become a lot less wasteful. Right now crawlers spend a lot of effort rediscovering the same structure over and over. It also raises interesting questions about whether sites will eventually provide different views for hum…

> It also raises interesting questions about whether sites will eventually provide different views for humans vs. automated agents in a more formalized way.

This question raises an interesting question about if this would exacerbate supply chain injection attacks. Show the innocuous page to the human, another to the bot.

Re: Cloudflare crawl endpoint

#62

Post by fleebee » Wed, Mar 11, 2026, 1:50 AM UTC

babelfish wrote: post #26

Didn't they just throw a (very public) fit over Perplexity doing the exact same thing?

The most egregious thing Perplexity did was to straight up ignore robots.txt. Cloudflare promise not to do that, so if we take their word for it, it's a quite different setup.

That said, I'm not fan of letting users forge whatever user agents they please. Instead, AIUI to opt-out of getting crawled I have to look for the existence of certain request headers[1].

[1]: https://developers.cloudflare.com/browser-rendering/referenc...

Re: Cloudflare crawl endpoint

#63

Post by tjpnz » Wed, Mar 11, 2026, 1:53 AM UTC

Do I have the option to fill it with junk for LLMs?

Re: Cloudflare crawl endpoint

#64

Post by _heimdall » Wed, Mar 11, 2026, 1:53 AM UTC

Lasang wrote: post #60

The idea of exposing a structured crawl endpoint feels like a natural evolution of robots.txt and sitemaps. If more sites provided explicit machine-readable entry points for crawlers, indexing could become a lot less wasteful. Right now crawlers spend a lot of effort rediscovering the same structure over and over. It also raises interesting questions about whether sites will eventually provide different views for hum…

I expect that if we still used REST indexing would be even less wasteful.

I've found myself falling pretty hard on the side of making APIs work for humans and expecting LLM providers to optimize around that. I don't need an MCP for a CLI tool, for example, I just need a good man page or `--help` documentation.

Re: Cloudflare crawl endpoint

#65

Post by pqdbr » Wed, Mar 11, 2026, 2:00 AM UTC

Off-topic, but I'm having a terrible experience with Cloudflare and would love to know if someone could offer some help.

All of a sudden, about 1/3 of all traffic to our website is being routed via EWR (New York) - me included -, even tough all our users and our origin servers are in Brazil.

We pay for the Pro plan but support has been of no help: after 20 days of 'debugging' and asking for MTRs and traceroutes, they told us to contact Claro (which is the same as telling me to contact Verizon) because 'it's their fault'.

Re: Cloudflare crawl endpoint

#66

Post by fbrncci » Wed, Mar 11, 2026, 2:06 AM UTC

Awesome, so I no longer have to use Firecrawl or my own crawler to scrape entire websites for an agent? Especially when needing residential proxies to do so on Cloudflare protected sites? Why though?

Re: Cloudflare crawl endpoint

#67

Post by tgrowazay » Wed, Mar 11, 2026, 2:09 AM UTC

pqdbr wrote: post #65

Off-topic, but I'm having a terrible experience with Cloudflare and would love to know if someone could offer some help. All of a sudden, about 1/3 of all traffic to our website is being routed via EWR (New York) - me included -, even tough all our users and our origin servers are in Brazil. We pay for the Pro plan but support has been of no help: after 20 days of 'debugging' and asking for MTRs and traceroutes, they…

It is possible that Claro has a bad route that sends all traffic destined for Cloudflare through New York.

Re: Cloudflare crawl endpoint

#68

Post by rglover » Wed, Mar 11, 2026, 2:14 AM UTC

Lasang wrote: post #60

The idea of exposing a structured crawl endpoint feels like a natural evolution of robots.txt and sitemaps. If more sites provided explicit machine-readable entry points for crawlers, indexing could become a lot less wasteful. Right now crawlers spend a lot of effort rediscovering the same structure over and over. It also raises interesting questions about whether sites will eventually provide different views for hum…

I just do a query param to toggle to markdown/text if ?llm=true on a route. Easy pattern that's opt-in.

Re: Cloudflare crawl endpoint

#69

Post by freakynit » Wed, Mar 11, 2026, 2:15 AM UTC

fbrncci wrote: post #66

Awesome, so I no longer have to use Firecrawl or my own crawler to scrape entire websites for an agent? Especially when needing residential proxies to do so on Cloudflare protected sites? Why though?

I have tried theirs... they are NOT proxies.. that means majority of the popular sites actually block scraping... even if they are protected by cloudflare itself.

Re: Cloudflare crawl endpoint

#70

Post by coreq » Wed, Mar 11, 2026, 2:32 AM UTC

The big question here is this a verified-bot on the Cloudflare WAF? Didn't Google get into trouble for using their search engine user agent and IPs to feed Gemini in Europe?