Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
1–10 of 799 posts
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#2[flagged]
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#3>We conducted an experiment by querying Perplexity AI with questions about these domains, and discovered Perplexity was still providing detailed information regarding the exact content hosted on each of these restricted domains
Thats... less conclusive than I'd like to see, especially for a content marketing article that's calling out a company in particular. Specifically it's unclear on whether Perplexity was crawling (ie. systematically viewing every page on the site without the direction of a human), or simply retrieving content on behalf of the user. I think most people would draw a distinction between the two, and would at least agree the latter is more acceptable than the former.
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#4I find this problem quite difficult to solve:
1. If I as a human request a website, then I should be shown the content. Everyone agrees.
2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into modifying the software you run locally.
3. If I now go one step further and use an LLM to summarize content because the authentic presentation is so riddled with ads, JavaScript, and pop-ups, that the content becomes borderline unusable, then why would the LLM accessing the website on my behalf be in a different legal category as my Firefox web browser accessing the website on my behalf?
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#5I do not really get why user-agent blocking measures are despised for browsers but celebrated for agents?
It’s a different UI, sure, but there should be no discrimination towards it as there should be no discrimination towards, say, Links terminal browser, or some exotic Firefox derivative.
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#6If you put info on the web, it should be available to everyone or everything with access.
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#7I find this problem quite difficult to solve: 1. If I as a human request a website, then I should be shown the content. Everyone agrees. 2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into…
How about I open a proxy, replace all ads with my ads, redirect the content to you and we share the ad revenue?
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#8>We conducted an experiment by querying Perplexity AI with questions about these domains, and discovered Perplexity was still providing detailed information regarding the exact content hosted on each of these restricted domains Thats... less conclusive than I'd like to see, especially for a content marketing article that's calling out a company in particular. Specifically it's unclear on whether Perplexity was crawli…
> I think most people would draw a distinction between the two, and would at least agree the latter is more acceptable than the former.
No. I should be able to control which automated retrieval tools can scrape my site, regardless of who commands it.
We can play cat and mouse all day, but I control the content and I will always win: I can just take it down when annoyed badly enough. Then nobody gets the content, and we can all thank upstanding companies like Perplexity for that collapse of trust.
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#9I find this problem quite difficult to solve: 1. If I as a human request a website, then I should be shown the content. Everyone agrees. 2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into…
If the LLM were running this sort of thing at the user's explicit request this would be fine. The problem is training. Every AI startup on the planet right now is aggressively crawling everything that will let them crawl. The server isn't seeing occasional summaries from interested users, but thousands upon thousands of bots repeatedly requesting every link they can find as fast as they can.
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#10I find this problem quite difficult to solve: 1. If I as a human request a website, then I should be shown the content. Everyone agrees. 2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into…
Correct, it’s user hostile to dictate which software is allowed to see content.