111–120 of 194 posts
> Honors robots.txt directives, including crawl-delay Sounds pretty useless for any serious AI company
If your delay is 1s and you publish less than 60 updates a minute on average I can still get 100%. Most crawls are not that latency sensitive, certainly not the ai ones.
HFT bots, now that is an entirely different ballgame.
[flagged]
Crawl jobs per day 5 per day
Maximum pages per crawl 100 pages
Earlier quoted context omitted.
Their free DNS is only a small piece of the pie. The fact that 30%+ of the web relies on their caching services, routablility services and DDoS protection services is the main pull. Their DNS is only really for data collection and to front as "good will"
> The fact that 30%+ of the web relies on their caching services 30% of the web might use their caching services. 'Relies on' implies that it wouldn't work without them, which I doubt is the case. It might be the case for the biggest 1% of that 30%. But not the whole lot.
Last time Cloudflare went down, their dashboard was also unavailable, so you couldn't turn off their proxy service anyway.
[flagged]
I'm surprised that Cloudflare hasn't started hosting a pre-scraped version of websites that use Cloudflare's proxy - something like https://www.example.com/cdn-cgi/cached-contents.json They already have the website content in their cache, so why not just cut out the middle man of scraping services and API's like this and publish it? Obviously there's good reasons NOT to, but I am surprised they haven't started offeri…
It’s a bit more complicated than that. This is their product Browser Rendering, which runs a real browser that loads the page and executes JavaScript. It’s a bit more involved than a simple curl scraping.
I'm surprised that Cloudflare hasn't started hosting a pre-scraped version of websites that use Cloudflare's proxy - something like https://www.example.com/cdn-cgi/cached-contents.json They already have the website content in their cache, so why not just cut out the middle man of scraping services and API's like this and publish it? Obviously there's good reasons NOT to, but I am surprised they haven't started offeri…
Doesn't work for pages protected by cloudflare in my experience. What a shame, they could've produced the problem and sold the solution.
Earlier quoted context omitted.
So what’s the user agent for their bot? They don’t seem to specify the default in the docs and it looks like it’s user configurable. So yet another opt out bot which you need your web server to match on special behaviour to block
Isn't this covered here? https://developers.cloudflare.com/browser-rendering/referenc...
Earlier quoted context omitted.
So what’s the user agent for their bot? They don’t seem to specify the default in the docs and it looks like it’s user configurable. So yet another opt out bot which you need your web server to match on special behaviour to block
>So yet another opt out bot which you need your web server to match on special behaviour to block Given that malicious bots are allegedly spoofing real user agents, "another user agent you have to add to your list" seems like the least of your problems.