Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity AI is lying about their user agent

rknight.me

Post reply on HN
Print view View original

261–270 of 555 posts

Re: Perplexity AI is lying about their user agent

#261

Post by CobrastanJorji »

epolanski wrote: post #188
putlake wrote: post #175

Earlier quoted context omitted.

It's funny I posted the inverse of this. As a web publisher, I am fine with folks using my content to train their models because this training does not directly steal any traffic. It's the "train an AI by reading all the books in the world" analogy. But what Perplexity is doing when they crawl my content in response to a user question is that they are decreasing the probability that this user would come to by content…

> they are decreasing the probability that this user would come to by content (via Google, for example). Google has been providing summaries of stuff and hijacking traffic for ages. I kid you not, in the tourism sector this has been a HUGE issue, we have seen 50%+ decrease in views when they started doing it. We paid gazzilions to write quality content for tourists about the most different places just so Google could…

I'm curious about the tourism sector problem. In tourism, I would think the goal would be to promote a location. You want people to be able to easily discover the location, get information about it, and presumably arrange to travel to those locations. If Google gets the information to the users, but doesn't send the tourist to the website, is that harmful? Is it a problem of ads on the tourism website? Or is more of problem of the site creator demonstrating to the site purchaser that the purchase was worthwhile?

Re: Perplexity AI is lying about their user agent

#262

Post by lolinder »

alt227 wrote: post #243
lolinder wrote: post #190

Earlier quoted context omitted.

What is a "visit"? TFA demonstrates that they got a hit on their site, that's how they got the logs. Is it necessary to load the JavaScript for it to count as a visit? What if I access the site with noscript? Or is it only a visit if I see all your recommended content? I usually block those recommendations so that I don't get distracted from the article I actually came to read—is my visit a less legitimate visit than…

> TFA demonstrates that they got a hit on their site Whats stopping perplexity caching this info say for 24 hours, and then redisplaying it to the next few hundred people who request it?

Then they don't get the extra hits. So is that it—is a "visit" important because of the data that you're able to collect from the visit?

Does this place HN's rampant use of archive.md on the same moral footing as Perplexity?

Re: Perplexity AI is lying about their user agent

#263

Post by elicksaur »

buro9 wrote: post #177
lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

The companies will scrape and internalise the "customer asked for this" requests... and slowly turn the latter into the former, or just their own tool as the scraper. No, easier to just ask a simple question: Does the company respect the access rules communicated via a web standard? No? In that case hard deny access to that company. These companies don't need to be given an inch.

This is exactly the concern and there’s a lot of comments just completely ignoring it or willfully conflating.

Ad block isn’t the same problem because it doesn’t and can’t steal the creator’s data.

Re: Perplexity AI is lying about their user agent

#264

Post by zarathustreal »

I know it’s obvious but I’m going to state it anyway just for emphasis:

Do not put anything on the public-facing internet that you don’t intend for people to use freely. You’re literally providing a free download. That’s the nature of the web and it always has been.

Re: Perplexity AI is lying about their user agent

#265

Post by lofaszvanitt »

twinge wrote: post #122
Mathnerd314 wrote: post #16

Earlier quoted context omitted.

It's been debated at length, but to make it short: piracy is not theft, and everyone in the LLM space has been taking other people’s content and so far getting away with it (pending lawsuits notwithstanding).

Aereo, Napster, Grokster, Grooveshark, Megaupload, and TVEyes: they all thought the same thing. Where are they now?

They were massacred by well funded corps. Who is on the side of single joes?

Re: Perplexity AI is lying about their user agent

#266

Post by FireInsight »

s3r3nity wrote: post #131
Mathnerd314 wrote: post #16

Earlier quoted context omitted.

It's been debated at length, but to make it short: piracy is not theft, and everyone in the LLM space has been taking other people’s content and so far getting away with it (pending lawsuits notwithstanding).

[flagged]

Something not being stealing isn't the same as it not being able to hurt people or companies financially. Revenue lost due to copyright breach is not money stolen from you.

I pay my indie creators fairly; big companies is when I stop caring.

Re: Perplexity AI is lying about their user agent

#267

Post by denton-scratch »

lolinder wrote: post #190
baxtr wrote: post #183

Earlier quoted context omitted.

What will happen if: Website owners decide to stop publishing because it’s not rewarded by a real human visit anymore? Then perplexity and the like won’t have new information to train their models on and no sites to answer the questions. I think there is a real content dilemma here at work. The incentives of Google and website owners were more or less aligned. This is not the case with perplexity.

What is a "visit"? TFA demonstrates that they got a hit on their site, that's how they got the logs. Is it necessary to load the JavaScript for it to count as a visit? What if I access the site with noscript? Or is it only a visit if I see all your recommended content? I usually block those recommendations so that I don't get distracted from the article I actually came to read—is my visit a less legitimate visit than…

> What exactly is Perplexity doing here that isn't okay that people don't already do with their local user agents?

It's in the title of TFA: they're being dishonest about who they are. PerplexityBot seems to understand that robots.txt is addressed to it.

It's understood that site operators have a right to use the User-Agent to discriminate among visitors; that's why robots.txt is a standard. Crawlers that disrespect the standard have for many years been considered beyond the pale; thieves and snoopers. TFA's complaint is entirely justified.

Re: Perplexity AI is lying about their user agent

#268

Post by luaKmua »

itsoktocry wrote: post #257
luaKmua wrote: post #245

Earlier quoted context omitted.

That line of thinking makes no sense. If the "content" had no value, why would google go through the effort of scraping it and presenting it to the user?

> If the "content" had no value, why would google go through the effort of scraping it and presenting it to the user? They don't present it all, they summarize it. And let's be serious here, I was being polite because I don't know the OPs business. But 99% of this sort of content is SEO trash and contributes to the wasteland that the internet is becoming. Feel free to point me to the good stuff.

Pedantry aside, let's restate as "present the core thoughts" to the user, which still implies value. I agree that most of google front page results are SEO garbage these days, but that's a separate issue from claiming that are summary of a piece of information removes the original of its value. I'd even argue that it transfers it from one entity to the other in this case.

Re: Perplexity AI is lying about their user agent

#269

Post by marcus0x62 »

nradov wrote: post #237
lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

Why should it be possible to stop an LLM from training itself on your data? If you want to restrict access to data then don't post it on a public website. It's easy enough to require registration and agreement to licensing terms for access. It seems like some website owners want to have their cake and eat it too. They want their content indexed by Google and other crawlers in order to drive search traffic but they do…

Because if I run a server - at my own expense - I get to use information provided by the client to determine what, if any, response to provide? This isn’t a very difficult concept to grasp.

Re: Perplexity AI is lying about their user agent

#270

Post by mattigames »

JimDabell wrote: post #186
maxrmk wrote: post #84

The author has misunderstood when the perplexity user agent applies. Web site owners shouldn’t dictate what browser users can access their site with - whether that’s chrome, firefox, or something totally different like perplexity. When retrieving a web page _for the user_ it’s appropriate to use a UA string that looks like a browser client. If perplexity is collecting training data in bulk without using their UA that…

Just to go a little bit more into detail on this, because the article and most of the conversation here is based on a big misunderstanding: robots.txt governs crawlers . Fetching a single user-specified URL is not crawling. Crawling is when you automatically follow links to continue fetching subsequent pages. Perplexity’s documentation that the article links to describes how their crawler works. That is not the piece…

Its fairly logical to assume that robots.txt governs robots (empahsis in "bots") not just crawlers, if they are only intended to block crawlers why aren't they called crawlers.txt instead and remove all ambiguity?
Post reply on HN