Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

blog.cloudflare.com

Post reply on HN
Print view View original

71–80 of 799 posts

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#71

Post by sejje »

randall wrote: post #54
bobbiechen wrote: post #24

Earlier quoted context omitted.

I like the terminology "crawler" vs. "fetcher" to distinguish between mass scraping and something more targeted as a user agent. I've been working on AI agent detection recently (see https://stytch.com/blog/introducing-is-agent/ ) and I think there's genuine value in website owners being able to identify AI agents to e.g. nudge them towards scoped access flows instead of fully impersonating a user with no controls. O…

A/ i love this distinction. B/ my brother used to use "fetcher" as a non-swear for "fucker"

He picked up that habit in Balmora.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#72

Post by blacksmith_tb »

hansvm wrote: post #52
yojo wrote: post #37

Earlier quoted context omitted.

Ads are a problematic business model, and I think your point there is kind of interesting. But AI companies disintermediating content creators from their users is NOT the web I want to replace it with. Let’s imagine you have a content creator that runs a paid newsletter. They put in lots of effort to make well-researched and compelling content. They give some of it away to entice interested parties to their site, whe…

Ofttimes people are sufficiently anti-ad that this point won't resonate well. I'm personally mostly in that camp in that with relatively few exceptions money seems to make the parts of the web I care about worse (it's hard to replace passion, and wading through SEO-optimized AI drivel to find a good site is a lot of work). Giving them concrete examples of sites which would go away can help make your point. E.g., Shel…

Sheldon died in 2008, but there's no doubt that all the bicycling wisdom he posted lives on!

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#73

Post by ai-christianson »

fxtentacle wrote: post #4

I find this problem quite difficult to solve: 1. If I as a human request a website, then I should be shown the content. Everyone agrees. 2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into…

Websites should be able to request payment. Who cares if it is a human or an agent of a human if it is paying for the request?

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#75

Post by Vinnl »

randall wrote: post #54
bobbiechen wrote: post #24

Earlier quoted context omitted.

I like the terminology "crawler" vs. "fetcher" to distinguish between mass scraping and something more targeted as a user agent. I've been working on AI agent detection recently (see https://stytch.com/blog/introducing-is-agent/ ) and I think there's genuine value in website owners being able to identify AI agents to e.g. nudge them towards scoped access flows instead of fully impersonating a user with no controls. O…

A/ i love this distinction. B/ my brother used to use "fetcher" as a non-swear for "fucker"

Did you tell him to stop trying to make fetcher happen?

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#76

Post by JimDabell »

cutemonster wrote: post #61
sbarre wrote: post #41

Earlier quoted context omitted.

All of these scenarios assume you have an unconditional right to access the content on a website in whatever way you want. Do you think you do? Or is there a balance between the owner's rights, who bears the content production and hosting/serving costs, and the rights of the end user who wishes to benefit from that content? If you say that you have the right, and that right should be legally protected, to do whatever…

If there's an article you want to read, and the ToS says that in between reading each paragraph, you must switch to their YouTube channel and look at their ads about cat food for 5 minutes, are your going to do that?

Hacker News has collectively answered this question by consistently voting up the archive.is links in the comments of every paywalled article posted here.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#77

Post by vertoc »

hansvm wrote: post #52
yojo wrote: post #37

Earlier quoted context omitted.

Ads are a problematic business model, and I think your point there is kind of interesting. But AI companies disintermediating content creators from their users is NOT the web I want to replace it with. Let’s imagine you have a content creator that runs a paid newsletter. They put in lots of effort to make well-researched and compelling content. They give some of it away to entice interested parties to their site, whe…

Ofttimes people are sufficiently anti-ad that this point won't resonate well. I'm personally mostly in that camp in that with relatively few exceptions money seems to make the parts of the web I care about worse (it's hard to replace passion, and wading through SEO-optimized AI drivel to find a good site is a lot of work). Giving them concrete examples of sites which would go away can help make your point. E.g., Shel…

But even your example gets worse with AI potentially - the "upsell" of his blog isn't paid posts but more subscribers so there will be thankful readers, a few donators, people talking about it. If the only interface becomes an AI summary of his work without credit, it's much more likely he stops writing as it'll seem like he's just screaming into the void

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#78

Post by Fraterkes »

gruez wrote: post #38
nnx wrote: post #5

I do not really get why user-agent blocking measures are despised for browsers but celebrated for agents? It’s a different UI, sure, but there should be no discrimination towards it as there should be no discrimination towards, say, Links terminal browser, or some exotic Firefox derivative.

>I do not really get why user-agent blocking measures are despised for browsers but celebrated for agents? AI broke the brains of many people. The internet isn't a monolith, but prior to the AI boom you'd be hard pressed to find people who were pro-copyright (except maybe a few who wanted to use it to force companies to comply with copyleft obligations), pro user-agent restrictions, or anti-scraping. Now such positio…

I think the intelligent conclusion would be that the people you are looking at have more nuanced beliefs than you initially thought. Talking about broken brains is often just mediocre projecting

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#80

Post by cj »

jasonjmcghee wrote: post #63
fxtentacle wrote: post #4

I find this problem quite difficult to solve: 1. If I as a human request a website, then I should be shown the content. Everyone agrees. 2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into…

I think it's an issue of scale. The next step in your progression here might be: If / when people have personal research bots that go and look for answers across a number of sites, requesting many pages much faster than humans do - what's the tipping point? Is personal web crawling ok? What if it gets a bit smarter and tried to anticipate what you'll ask and does a bunch of crawling to gather information regularly to…

Doesn't o3 sort of already do this? Whenever I ask it something, it makes it look like it simultaneously opens 3-8 pages (something a human can't do).

Seems like a reasonable stance would be something like "Following the no crawl directive is especially necessary when navigating websites faster than humans can."

> What if it gets a bit smarter and tried to anticipate what you'll ask and does a bunch of crawling to gather information regularly to try to stay up to date on things (from your machine)?

To be fair, Google Chrome already (somewhat) does this by preloading links it thinks you might click, before you click it.

But your point is still valid. We tolerate it because as website owners, we want our sites to load fast for users. But if we're just serving pages to robots and the data is repackaged to users without citing the original source, then yea... let's rethink that.

Post reply on HN