Really hard to understand costs here. What is a reasonable pages per second? Should I assume with politeness that I'm basically at 1 page per second == 3600 pages/hour? Seems painfully slow.
Cloudflare crawl endpoint
31–40 of 194 posts
Re: Cloudflare crawl endpoint
#32I'm surprised that Cloudflare hasn't started hosting a pre-scraped version of websites that use Cloudflare's proxy - something like https://www.example.com/cdn-cgi/cached-contents.json They already have the website content in their cache, so why not just cut out the middle man of scraping services and API's like this and publish it? Obviously there's good reasons NOT to, but I am surprised they haven't started offeri…
> I'm surprised that Cloudflare hasn't started hosting a pre-scraped version of websites that use Cloudflare's proxy It's entirely possible that they're doing this under the hood for cases where they can clearly identify the content they have cached is public.
Based on the post, it seems likely that they'd just delay per the robots.txt policy no matter what, and do a full browser render of the cached page to get the content. Probably overkill for lots and lots of sites. An HTML fetch + readability is really cheap.
Re: Cloudflare crawl endpoint
#33[flagged]
Re: Cloudflare crawl endpoint
#34[flagged]
Re: Cloudflare crawl endpoint
#35All what was expected, first they do a huge campaign to out evil scrapers. We should use their service to ensure your website block LLMs and bots to come scraping them. Look how bad it is. And once that is well setup, and they have their walled garden, then they can present their own API to scrape websites. All well done to be used by your LLM. But as you know, they are the gate keeper so that the Mafia boss decide w…
Re: Cloudflare crawl endpoint
#36[flagged]
Re: Cloudflare crawl endpoint
#37Will this crawler be run behind or infront of their bot blocker logic?
Re: Cloudflare crawl endpoint
#38Is cloudflare becoming a mob outfit? Because they are selling scraping countermeasures but are now selling scraping too. And they can pull it off because of their reach over the internet with the free DNS.
Re: Cloudflare crawl endpoint
#39I'm surprised that Cloudflare hasn't started hosting a pre-scraped version of websites that use Cloudflare's proxy - something like https://www.example.com/cdn-cgi/cached-contents.json They already have the website content in their cache, so why not just cut out the middle man of scraping services and API's like this and publish it? Obviously there's good reasons NOT to, but I am surprised they haven't started offeri…
Not the same thing, but they have something close (it's not on-by-default, yet) [1]:
> Cloudflare's network now supports real-time content conversion at the source, for enabled zones using content negotiation headers. Now when AI systems request pages from any website that uses Cloudflare and has Markdown for Agents enabled, they can express the preference for text/markdown in the request. Our network will automatically and efficiently convert the HTML to markdown, when possible, on the fly.
Re: Cloudflare crawl endpoint
#40Instead of "should have been an email" this is "should have been a prompt" and can be run locally instead. There are a number of ways to do this from a linux terminal.
``` write a custom crawler that will crawl every page on a site (internal links to the original domain only, scroll down to mimic a human, and save the output as a WebP screenshot, HTML, Markdown, and structured JSON. Make it designed to run locally in a terminal on a linux machine using headless Google Chrome and take advantage of multiple cores to run multiple pages simultaneously while keeping in mind that it might have to throttle if the server gets hit too fast from the same IP. ```
Might use available open source software such as python, playwright, beautifulsoup4, pillow, aiofiles, trafilatura