Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity AI is lying about their user agent

rknight.me

Post reply on HN
Print view View original

181–190 of 555 posts

Re: Perplexity AI is lying about their user agent

#181

Post by Terr_ »

brookst wrote: post #41
Mathnerd314 wrote: post #16

Earlier quoted context omitted.

It's been debated at length, but to make it short: piracy is not theft, and everyone in the LLM space has been taking other people’s content and so far getting away with it (pending lawsuits notwithstanding).

If using copyrighted material to train an LLM is theft, so is reading a book.

If the LLM is automatically equivalent to a human doing the same task, that means it's even worse: The companies are guilty of slavery. With children.

It also means reworking patent law, which holds that you can't just throw "with a computer" onto something otherwise un-patentable.

Clearly, there are other factors to consider, such as scope, intended purpose, outcome...

Re: Perplexity AI is lying about their user agent

#182

Post by LeifCarrotson »

putlake wrote: post #153

A lot of comments here are confusing the two use cases for crawling: training and summarization. Perplexity's utility as an answer engine is RAG (retrieval augmented generation). In response to your question, they search the web, crawl relevant URLs and summarize them. They do include citations in their response to the user, but in practice no one clicks through on the tiny (1), (2) links to go to the source. So if y…

Google and Bing increasingly do the same thing with their answer box featured snippets.

Re: Perplexity AI is lying about their user agent

#183

Post by baxtr »

lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

What will happen if:

Website owners decide to stop publishing because it’s not rewarded by a real human visit anymore?

Then perplexity and the like won’t have new information to train their models on and no sites to answer the questions.

I think there is a real content dilemma here at work. The incentives of Google and website owners were more or less aligned.

This is not the case with perplexity.

Re: Perplexity AI is lying about their user agent

#184

Post by lolinder »

buro9 wrote: post #177
lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

The companies will scrape and internalise the "customer asked for this" requests... and slowly turn the latter into the former, or just their own tool as the scraper. No, easier to just ask a simple question: Does the company respect the access rules communicated via a web standard? No? In that case hard deny access to that company. These companies don't need to be given an inch.

> Does the company respect the access rules communicated via a web standard? No? In that case hard deny access to that company.

So should Firefox not allow changing the user agent in order to bypass websites that erroneously claim to not work on Firefox?

Re: Perplexity AI is lying about their user agent

#185

Post by maxrmk »

rknightuk wrote: post #108
maxrmk wrote: post #84

The author has misunderstood when the perplexity user agent applies. Web site owners shouldn’t dictate what browser users can access their site with - whether that’s chrome, firefox, or something totally different like perplexity. When retrieving a web page _for the user_ it’s appropriate to use a UA string that looks like a browser client. If perplexity is collecting training data in bulk without using their UA that…

It’s not retrieving a web page though is it? It’s retrieving the content then manipulating it. Perplexity isn’t a web browser.

I’d consider it a web browser but that’s a vague enough term that I can understand seeing it differently.

I’d be disappointed if it became common to block clients like this though. To me this feels like blocking google chrome because you don’t want to show up in google search (which is totally fine to want, for the record). Unnecessarily user hostile because you don’t approve of the company behind the client.

Re: Perplexity AI is lying about their user agent

#186

Post by JimDabell »

maxrmk wrote: post #84

The author has misunderstood when the perplexity user agent applies. Web site owners shouldn’t dictate what browser users can access their site with - whether that’s chrome, firefox, or something totally different like perplexity. When retrieving a web page _for the user_ it’s appropriate to use a UA string that looks like a browser client. If perplexity is collecting training data in bulk without using their UA that…

Just to go a little bit more into detail on this, because the article and most of the conversation here is based on a big misunderstanding:

robots.txt governs crawlers. Fetching a single user-specified URL is not crawling. Crawling is when you automatically follow links to continue fetching subsequent pages.

Perplexity’s documentation that the article links to describes how their crawler works. That is not the piece of software that fetches individual web pages when a user asks for them. That’s just a regular user-agent, because it’s acting as an agent for the user.

The distinction between crawling and not crawling has been very firmly established for decades. You can see it in action with wget. If you fetch a specific URL with `wget https://www.example.com` then wget will just fetch that URL. It will not fetch robots.txt at all.

If you tell wget to act recursively with `wget --recursive https://www.example.com` to crawl that website, then wget will fetch `https://www.example.com`, look for links on the page, then if it finds any links to other pages, it will fetch `https://www.example.com/robots.txt` to check if it is permitted to fetch any subsequent links.

This is the difference between fetching a web page and crawling a website. Perplexity is following the very well established norms here.

Re: Perplexity AI is lying about their user agent

#187

Post by mrweasel »

wrs wrote: post #3

I don’t think we should lump together “AI company scraping a website to train their base model” and “AI tool retrieving a web page because I asked it to”. At least, those should be two different user agents so you have the option to block one and not the other.

Personally I don't even think that the issue. I'd prefer correct user-agent, that just common decency and shouldn't be an issue for most.

What I do expect the AI companies to do is to check the license of the content they scrape and follow that. Let's say I run a blog, and I have a CC BY-NC 4.0 license. You can train your AI and that content, as long as it's non-commercial. Otherwise you'd need to contact me an negotiate and appropriate license, for a fee. Or you can train your AI on my personal Github repo, where everything is ISC, that's fine, but for my work, which is GPLv3, then you have to ensure that the code your LLM returns is also under the GPLv3. Does any of the AI companies check that the license of ANYTHING?

Re: Perplexity AI is lying about their user agent

#188

Post by epolanski »

putlake wrote: post #175
lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

It's funny I posted the inverse of this. As a web publisher, I am fine with folks using my content to train their models because this training does not directly steal any traffic. It's the "train an AI by reading all the books in the world" analogy. But what Perplexity is doing when they crawl my content in response to a user question is that they are decreasing the probability that this user would come to by content…

> they are decreasing the probability that this user would come to by content (via Google, for example).

Google has been providing summaries of stuff and hijacking traffic for ages.

I kid you not, in the tourism sector this has been a HUGE issue, we have seen 50%+ decrease in views when they started doing it.

We paid gazzilions to write quality content for tourists about the most different places just so Google could put it on their homepage.

It's just depressing. I'm more and more convinced that the age of regulations and competition is gone, US does want to have unkillable monopolies in the tech sector and we are all peons.

Re: Perplexity AI is lying about their user agent

#189

Post by skeledrew »

rknightuk wrote: post #140
skeledrew wrote: post #119

I really don't see this as that big of an issue with Perplexity per se, as sources are cited in the content. Users can still opt to visit relevant sources.

But they don’t need to because it’s been summarised already. So I get nothing while perplexity make more money.

It's not them though, but the users of their services. Like one thing I find myself doing a lot of to get through long reading lists is "summarize in detail X", where X is the URL of an article, etc. Depending on the result I may decide to actually read the entire thing or move on. Somewhere down the road it would be nice to just provide that list and have an AI order by how relevant the content is to me at that particular point in time...

Re: Perplexity AI is lying about their user agent

#190

Post by lolinder »

baxtr wrote: post #183
lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

What will happen if: Website owners decide to stop publishing because it’s not rewarded by a real human visit anymore? Then perplexity and the like won’t have new information to train their models on and no sites to answer the questions. I think there is a real content dilemma here at work. The incentives of Google and website owners were more or less aligned. This is not the case with perplexity.

What is a "visit"? TFA demonstrates that they got a hit on their site, that's how they got the logs.

Is it necessary to load the JavaScript for it to count as a visit? What if I access the site with noscript?

Or is it only a visit if I see all your recommended content? I usually block those recommendations so that I don't get distracted from the article I actually came to read—is my visit a less legitimate visit than other people's?

What exactly is Perplexity doing here that isn't okay that people don't already do with their local user agents?

Post reply on HN