Perplexity is using stealth, undeclared crawlers to evade no-crawl directives • Hacker News Board

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#61

Post by cutemonster » Mon, Aug 04, 2025, 2:43 PM UTC

sbarre wrote: post #41

fxtentacle wrote: post #4

I find this problem quite difficult to solve: 1. If I as a human request a website, then I should be shown the content. Everyone agrees. 2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into…

All of these scenarios assume you have an unconditional right to access the content on a website in whatever way you want. Do you think you do? Or is there a balance between the owner's rights, who bears the content production and hosting/serving costs, and the rights of the end user who wishes to benefit from that content? If you say that you have the right, and that right should be legally protected, to do whatever…

If there's an article you want to read, and the ToS says that in between reading each paragraph, you must switch to their YouTube channel and look at their ads about cat food for 5 minutes, are your going to do that?

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#62

Post by gruez » Mon, Aug 04, 2025, 2:43 PM UTC

sbarre wrote: post #41

fxtentacle wrote: post #4

I find this problem quite difficult to solve: 1. If I as a human request a website, then I should be shown the content. Everyone agrees. 2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into…

All of these scenarios assume you have an unconditional right to access the content on a website in whatever way you want. Do you think you do? Or is there a balance between the owner's rights, who bears the content production and hosting/serving costs, and the rights of the end user who wishes to benefit from that content? If you say that you have the right, and that right should be legally protected, to do whatever…

>Or is there a balance between the owner's rights, who bears the content production and hosting/serving costs, and the rights of the end user who wishes to benefit from that content?

If you believe in this principle, fair enough, but are you going to apply this consistently? If it's fair game for a blog to restrict access to AI agents, what does that mean for other user agents that companies disagree with, like browsers with adblock? Does it just boil down to "it's okay if a person does it but not okay if a big evil corporation does it?"

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#63

Post by jasonjmcghee » Mon, Aug 04, 2025, 2:43 PM UTC

fxtentacle wrote: post #4

I find this problem quite difficult to solve: 1. If I as a human request a website, then I should be shown the content. Everyone agrees. 2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into…

I think it's an issue of scale.

The next step in your progression here might be:

If / when people have personal research bots that go and look for answers across a number of sites, requesting many pages much faster than humans do - what's the tipping point? Is personal web crawling ok? What if it gets a bit smarter and tried to anticipate what you'll ask and does a bunch of crawling to gather information regularly to try to stay up to date on things (from your machine)? Or is it when you tip the scale further and do general / mass crawling for many users to consume that it becomes a problem?

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#64

Post by lukeschlather » Mon, Aug 04, 2025, 2:44 PM UTC

johnfn wrote: post #39

fxtentacle wrote: post #4

I find this problem quite difficult to solve: 1. If I as a human request a website, then I should be shown the content. Everyone agrees. 2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into…

Unless I am misunderstanding you, you are talking about something different than the article. The article is talking about web-crawling. You are talking about local / personal LLM usage. No one has any problems with local / personal LLM usage. It's when Perplexity uses web crawlers that an issue arises.

You probably need a computer that costs $250,000 or more to run the kind of LLM that Perplexity uses, but with batching it costs pennies to have the same LLM fetch a page for you, summarize the content, and tell you what is on it. And the power usage similarly, running the LLM for a single user will cost you a huge amount of money relative to the power it takes in a cloud environment with many users.

Perplexity's "web crawler" is mostly operating like this on behalf of users, so they don't need a massively expensive computer to run an LLM.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#65

Post by fluidcruft » Mon, Aug 04, 2025, 2:44 PM UTC

gruez wrote: post #50

fluidcruft wrote: post #33

Earlier quoted context omitted.

If the AI archives/caches all the results it accesses and enough people use it, doesn't it become a scraper? Just learn off the cached data. Being the man-in-the-middle seems like a pretty easy way to scrape salient content while also getting signals about that content's value.

>If the AI archives/caches all the results it accesses and enough people use it, doesn't it become a scraper? That's basically how many crowdsourced crawling/archive projects work. For instance, sci-hub and RECAP[1]. Do you think they should be shut down as well? In both cases there's even a stronger justification to shutting them down, because the original content is paywalled and you could plausibly argue there's l…

I didn't suggest Perplexity should be shut down, though. And yes, in your analogy sites are completely justified to take whatever actions they can to block people who are building those caches.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#66

Post by wulfstan » Mon, Aug 04, 2025, 2:44 PM UTC

JimDabell wrote: post #26

Their test seems flawed: > We created multiple brand-new domains, similar to testexample.com and secretexample.com. These domains were newly purchased and had not yet been indexed by any search engine nor made publicly accessible in any discoverable way. We implemented a robots.txt file with directives to stop any respectful bots from accessing any part of a website: > We conducted an experiment by querying Perplexit…

Yeah I'm not so sure about that.

If Perplexity are visiting that page on your behalf to give you some information and aren't doing anything else with it, and just throw away that data afterwards, then you may have a point. As a site owner, I feel it's still my decision what I do and don't let you do, because you're visiting a page that I own and serve.

But if, as I suspect, Perplexity are visiting that page and then using information from that webpage in order to train their model then sorry mate, you're a crawler, you're just using a user as a proxy for your crawling activity.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#67

Post by Spacecosmonaut » Mon, Aug 04, 2025, 2:45 PM UTC

fxtentacle wrote: post #4

I find this problem quite difficult to solve: 1. If I as a human request a website, then I should be shown the content. Everyone agrees. 2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into…

Regarding point 3: The problem from the perspective of websites would not be any different if they had been completely ad free. People would still consume LLM generated summaries because they cut down clicks and eyeballing to present you information that directly pertains to the promt.

The whole concept of a "website" will simply become niche. How many zoomers still visit any but the most popular websites?

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#68

Post by [deleted] » Mon, Aug 04, 2025, 2:46 PM UTC

gruez wrote: post #3

>We conducted an experiment by querying Perplexity AI with questions about these domains, and discovered Perplexity was still providing detailed information regarding the exact content hosted on each of these restricted domains Thats... less conclusive than I'd like to see, especially for a content marketing article that's calling out a company in particular. Specifically it's unclear on whether Perplexity was crawli…

[deleted]

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#69

Post by lukeschlather » Mon, Aug 04, 2025, 2:47 PM UTC

cjonas wrote: post #42

hombre_fatal wrote: post #21

Earlier quoted context omitted.

TFA isn’t talking about crawling to harvest training data. It’s talking about Perplexity crawling sites on demand in response to user queries and then complaining that no it’s not fine, hence this thread.

Doesn't perplexity crawl to harvest and index data like a traditional search engine? Or is it all "on demand"?

For the most part I would assume they pay for access to Google or Bing's index. I also assume they don't really train models. So all their "crawling" is on behalf of users.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#70

Post by bee_rider » Mon, Aug 04, 2025, 2:49 PM UTC

yojo wrote: post #37

fxtentacle wrote: post #4

I find this problem quite difficult to solve: 1. If I as a human request a website, then I should be shown the content. Everyone agrees. 2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into…

Ads are a problematic business model, and I think your point there is kind of interesting. But AI companies disintermediating content creators from their users is NOT the web I want to replace it with. Let’s imagine you have a content creator that runs a paid newsletter. They put in lots of effort to make well-researched and compelling content. They give some of it away to entice interested parties to their site, whe…

I think it’s basically impossible to prevent AI crawlers. It is like video game cheating, at the extreme they could literally point a camera at the screen and have it do image processing, and talk to the computer through the USB port emulating, a mouse and keyboard outside the machine. They don’t do that, of course, because it is much easier to do it all in software, but that is the ultimate circumvention of any attempt to block them out that doesn’t also block out humans.

I think the business model for “content creating” is going have to change, for better or worse (a lot of YouTube stars are annoying as hell, but sure, stuff like well-written news and educational articles falls under this umbrella as well, so it is unfortunate that they will probably be impacted too).