Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare crawl endpoint

developers.cloudflare.com

Post reply on HN
Print view View original

71–80 of 194 posts

Re: Cloudflare crawl endpoint

#71

Post by coreq »

janalsncm wrote: post #42
michaelmior wrote: post #15

Earlier quoted context omitted.

> I'm surprised that Cloudflare hasn't started hosting a pre-scraped version of websites that use Cloudflare's proxy It's entirely possible that they're doing this under the hood for cases where they can clearly identify the content they have cached is public.

How would they know the content hasn’t changed without hitting the website?

They wouldn't, well there's Etag and alike but it still a round trip on level 7 to the origin. However the pattern generally is to say when the content is good to in the Response headers, and cache on that duration, for an example a bitcoin pricing aggregator might say good for 60 seconds (with disclaimers on page that this isn't market data), whilst My Little Town news might say that an article is good for an hour (to allow Updates) and the homepage is good for 5 minutes to allow breaking news article to not appear too far behind.

Re: Cloudflare crawl endpoint

#72

Post by pdntspa »

Lasang wrote: post #60

The idea of exposing a structured crawl endpoint feels like a natural evolution of robots.txt and sitemaps. If more sites provided explicit machine-readable entry points for crawlers, indexing could become a lot less wasteful. Right now crawlers spend a lot of effort rediscovering the same structure over and over. It also raises interesting questions about whether sites will eventually provide different views for hum…

They already do...

A lot of known crawlers will get a crawler-optimized version of the page

Re: Cloudflare crawl endpoint

#73

Post by rafram »

pdntspa wrote: post #72
Lasang wrote: post #60

The idea of exposing a structured crawl endpoint feels like a natural evolution of robots.txt and sitemaps. If more sites provided explicit machine-readable entry points for crawlers, indexing could become a lot less wasteful. Right now crawlers spend a lot of effort rediscovering the same structure over and over. It also raises interesting questions about whether sites will eventually provide different views for hum…

They already do... A lot of known crawlers will get a crawler-optimized version of the page

Do they? AFAIK Google forbids that, and they’ll occasionally test that you aren’t doing it.

Re: Cloudflare crawl endpoint

#74

Post by pdntspa »

rafram wrote: post #73
pdntspa wrote: post #72

Earlier quoted context omitted.

They already do... A lot of known crawlers will get a crawler-optimized version of the page

Do they? AFAIK Google forbids that, and they’ll occasionally test that you aren’t doing it.

I haven't checked in a while but I know for a fact that Amazon does or did it

Re: Cloudflare crawl endpoint

#75

Post by OptionOfT »

janalsncm wrote: post #42
michaelmior wrote: post #15

Earlier quoted context omitted.

> I'm surprised that Cloudflare hasn't started hosting a pre-scraped version of websites that use Cloudflare's proxy It's entirely possible that they're doing this under the hood for cases where they can clearly identify the content they have cached is public.

How would they know the content hasn’t changed without hitting the website?

Caching headers?

(Which, on Akamai, are by default ignored!)

Re: Cloudflare crawl endpoint

#76

Post by oefrha »

shadowfiend wrote: post #38
ljm wrote: post #10

Is cloudflare becoming a mob outfit? Because they are selling scraping countermeasures but are now selling scraping too. And they can pull it off because of their reach over the internet with the free DNS.

No: https://developers.cloudflare.com/browser-rendering/rest-api...

That's not the perfect defense you think it is. Plenty of robots.txts[1] technically allow scraping their main content pages as long as your user-agent isn't explicitly disallowed, but in practice they're behind Cloudflare so they still throw up Cloudflare bot check if you actually attempt to crawl.

And forget about crawling. If you have a less reputable IP (basically every IP in third world countries are less reputable, for instance), you can be CAPTCHA'ed to no end by Cloudflare even as a human user, on the default setting, so plenty of site owners with more reputable home/office IPs don't even know what they subject a subset of their users to.

[1] E.g. https://www.wired.com/robots.txt to pick an example high up on HN front page.

Re: Cloudflare crawl endpoint

#77

Post by Dylan16807 »

Retr0id wrote: post #13
ljm wrote: post #10

Is cloudflare becoming a mob outfit? Because they are selling scraping countermeasures but are now selling scraping too. And they can pull it off because of their reach over the internet with the free DNS.

For a long time cloudflare has proudly protected DDoS-as-a-service sites (but of course, they claim they don't "host" them)

Are you using the word "claim" to call them wrong or for a more confusing reason?

Because I'm pretty sure they are not in fact wrong.

Re: Cloudflare crawl endpoint

#78

Post by pocksuppet »

ljm wrote: post #10

Is cloudflare becoming a mob outfit? Because they are selling scraping countermeasures but are now selling scraping too. And they can pull it off because of their reach over the internet with the free DNS.

Was it ever not one? They protect a lot of DDoS-for-hire sites from DDoS by their competitors. In return they increase the quantity of DDoS on the internet. They offer you a service for $150, then months later suddenly demand $150k in 24 hours or they shut down your business. If you use them as a DNS registrar they will hold your domain hostage.

Re: Cloudflare crawl endpoint

#80

Post by pocksuppet »

Lasang wrote: post #60

The idea of exposing a structured crawl endpoint feels like a natural evolution of robots.txt and sitemaps. If more sites provided explicit machine-readable entry points for crawlers, indexing could become a lot less wasteful. Right now crawlers spend a lot of effort rediscovering the same structure over and over. It also raises interesting questions about whether sites will eventually provide different views for hum…

Apart from the obvious problem: presenting something different to crawlers and humans.
Post reply on HN