Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare crawl endpoint

developers.cloudflare.com

Post reply on HN
Print view View original

81–90 of 194 posts

Re: Cloudflare crawl endpoint

#81

Post by 6510 »

rafram wrote: post #73
pdntspa wrote: post #72

Earlier quoted context omitted.

They already do... A lot of known crawlers will get a crawler-optimized version of the page

Do they? AFAIK Google forbids that, and they’ll occasionally test that you aren’t doing it.

With google covering only 3% I wonder how much people still care and if they should. Funny: I own and know sites that are by far the best resource on the topic but shouldn't have so many links google says. It's like I ask you for a page about cuban chains then you say you don't have it because they had to many links. Or your greengrocer suddenly doesn't have apples because his supplier now offers more than 5 different kinds so he will never buy there again.

Re: Cloudflare crawl endpoint

#83

Post by Retr0id »

Dylan16807 wrote: post #77
Retr0id wrote: post #13

Earlier quoted context omitted.

For a long time cloudflare has proudly protected DDoS-as-a-service sites (but of course, they claim they don't "host" them)

Are you using the word "claim" to call them wrong or for a more confusing reason? Because I'm pretty sure they are not in fact wrong.

The distinction between a caching proxy and an origin server is pretty meaningless when you're serving static content, if you ask me.

Re: Cloudflare crawl endpoint

#84

Post by arjie »

echoangle wrote: post #56
arjie wrote: post #49

Oh man, I was hoping I could offer a nicely-crawled version of my site. It would be cool if they offered that for site admins. Then everyone who wanted to crawl would just get a thing they could get for pure transfer cost. I suppose I could build one by submitting a crawl job against myself and then offering a `static.` subdomain on each thing that people could access. Then it's pure HTML instant-load.

I don’t really get the usecase. Is your site static? Then you should just render it to html files and host the static files. And if it’s not static, how would a snapshot of the pages help if they change later? And also why not just add some caching to the site then?

Ah the use-case is archive.org but fast. But it's okay. Before I die I will make the static copy of my site myself.

Re: Cloudflare crawl endpoint

#85

Post by tempest_ »

tgrowazay wrote: post #67
pqdbr wrote: post #65

Off-topic, but I'm having a terrible experience with Cloudflare and would love to know if someone could offer some help. All of a sudden, about 1/3 of all traffic to our website is being routed via EWR (New York) - me included -, even tough all our users and our origin servers are in Brazil. We pay for the Pro plan but support has been of no help: after 20 days of 'debugging' and asking for MTRs and traceroutes, they…

It is possible that Claro has a bad route that sends all traffic destined for Cloudflare through New York.

Every once and a while we have had Bell Canada route a request that should be going about 6 blocks away across the continent and back.

They are not super helpful fixing it either.

Re: Cloudflare crawl endpoint

#86

Post by jen729w »

iso-logi wrote: post #18
ljm wrote: post #10

Is cloudflare becoming a mob outfit? Because they are selling scraping countermeasures but are now selling scraping too. And they can pull it off because of their reach over the internet with the free DNS.

Their free DNS is only a small piece of the pie. The fact that 30%+ of the web relies on their caching services, routablility services and DDoS protection services is the main pull. Their DNS is only really for data collection and to front as "good will"

> The fact that 30%+ of the web relies on their caching services

30% of the web might use their caching services. 'Relies on' implies that it wouldn't work without them, which I doubt is the case.

It might be the case for the biggest 1% of that 30%. But not the whole lot.

Re: Cloudflare crawl endpoint

#88

Post by Sebguer »

pocksuppet wrote: post #78
ljm wrote: post #10

Is cloudflare becoming a mob outfit? Because they are selling scraping countermeasures but are now selling scraping too. And they can pull it off because of their reach over the internet with the free DNS.

Was it ever not one? They protect a lot of DDoS-for-hire sites from DDoS by their competitors. In return they increase the quantity of DDoS on the internet. They offer you a service for $150, then months later suddenly demand $150k in 24 hours or they shut down your business. If you use them as a DNS registrar they will hold your domain hostage.

yeah, GP completely fails to realize that Cloudflare has always played both sides. that is their entire business model, and it was transparent from the beginning that they would absolutely do the same here.

Re: Cloudflare crawl endpoint

#89

Post by weird-eye-issue »

Imustaskforhelp wrote: post #41
ipaddr wrote: post #20

Earlier quoted context omitted.

"I didn't want to scrape/crawl it myself because I felt like it would feel like yet another scraping effort for AI and strain resources of developers" You feel better paying someone to do the same thimg?

I actually don't but it seems that cloudflare caches responses so if anything instead of straining the developer resources, it would strain more cloudflare resources and cloudflare could better handle that more efficiently with their own crawl product. Also, I am genuinely open to feedback (Like a lot) so just let me know if you know of any other alternative too for the particular thing that I wish to create and I wo…

This is used to scrape third-party sites not necessarily behind cloudflare so it has nothing to do with whether cloudflare caches it or not plus when using their browser rendering it doesn't even fetch cached responses anyways....

Re: Cloudflare crawl endpoint

#90

Post by isodev »

ljm wrote: post #10

Is cloudflare becoming a mob outfit? Because they are selling scraping countermeasures but are now selling scraping too. And they can pull it off because of their reach over the internet with the free DNS.

They always have been.

They also use their dominant position to apply political pressure when they don’t like how a country chooses to run things.

So yeah, we’ve created another mega corp monster that will hurt for years to come.

Post reply on HN