Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity AI is lying about their user agent

rknight.me

Post reply on HN
Print view View original

431–440 of 555 posts

Re: Perplexity AI is lying about their user agent

#431

Post by lolinder »

denton-scratch wrote: post #423
lolinder wrote: post #286

Earlier quoted context omitted.

> It's in the title of TFA: they're being dishonest about who they are. PerplexityBot seems to understand that robots.txt is addressed to it. First, I'm ignoring the output of Perplexity. I have no reason to believe that they gave the LLM any knowledge about its internal operations, it's just riffing off of what OP is saying. Second, PerplexityBot is the user agent that they use when crawling and indexing. They never…

> (which are notably not the same as crawling) Is that a distinction without a difference? I think the robots.txt RFC was addressed specifically to crawlers ; so technically "ad hoc" requests generated automatically (i.e. by robots) aren't included. But the distinction operators would like to make is between humans and automata. Whether some automaton is a crawler or not isn't relevant.

Actually, no, the fact that it's a crawler is the most important fact. The reason why website operators care at all about robots accessing their site (as distinct from humans controlling a browser) is historically one of two reasons:

* The pattern of requests can be very problematic. Impolite crawlers are totally capable of taking down a website by hitting it over and over and over again for hours in a way that humans won't.

* Crawlers are generally used to build search indexes, so instructing them about URLs that would be inappropriate to have show up in a search is relevant.

The behavior that OP is complaining about is that when the user pastes a URL into Perplexity, Perplexity fetches that URL. Neither the traffic pattern nor the persistence profile are remotely similar to typical crawler behavior. As far as I can see there's almost nothing to distinguish it from someone using Edge and then using Edge's built-in summarizer.

Re: Perplexity AI is lying about their user agent

#433

Post by Dylan16807 »

elicksaur wrote: post #263
buro9 wrote: post #177

Earlier quoted context omitted.

The companies will scrape and internalise the "customer asked for this" requests... and slowly turn the latter into the former, or just their own tool as the scraper. No, easier to just ask a simple question: Does the company respect the access rules communicated via a web standard? No? In that case hard deny access to that company. These companies don't need to be given an inch.

This is exactly the concern and there’s a lot of comments just completely ignoring it or willfully conflating. Ad block isn’t the same problem because it doesn’t and can’t steal the creator’s data.

If that's the concern, then ask for a line in the terms and conditions that explicitly says a user-initiated request will not be saved or used for training. Don't act like the access itself is an affront.

Re: Perplexity AI is lying about their user agent

#434

Post by Dylan16807 »

baxtr wrote: post #323
lolinder wrote: post #190

Earlier quoted context omitted.

What is a "visit"? TFA demonstrates that they got a hit on their site, that's how they got the logs. Is it necessary to load the JavaScript for it to count as a visit? What if I access the site with noscript? Or is it only a visit if I see all your recommended content? I usually block those recommendations so that I don't get distracted from the article I actually came to read—is my visit a less legitimate visit than…

A visit is a human reader. At the very least they get exposed to your website name. Notice your product/service if you get lucky. Become a customer at a later visit. We are talking about cutting the first step off so that everything which may come afterwards is cut off as well.

There was a human reader on the other side of the summarization feature. And they did get exposed to the website name. Is that not enough? Would it be different if equivalent summarization was being done by a browser extension?

Re: Perplexity AI is lying about their user agent

#435

Post by bshacklett »

mrweasel wrote: post #249
bshacklett wrote: post #210

Earlier quoted context omitted.

If Perplexity’s source code is downloaded from a public web site or other repository, and you take the time to understand the code and produce your own novel implementation, then yes. Now, if you “get it from a friend”, illegally, _or_ you just redeploy the code, without creating a transformative work, then there’s a problem. > Just pay the stupid license and if that makes your business unsustainable then it's not mu…

> They’d be fools to buy licenses before it’s been decided. They are willingly ignoring licenses until someone sues them? That's still illegal and completely immoral. There is tons of data to train on. The entirety of Wikipedia, all of StackOverflow (at least previously), all of the BSD and MIT licenses source code on Github, the entire Gutenberg project. So much stuff, freely and legally available, yet their feel th…

The legality of their behavior is not currently well defined, because it's unprecedented. Fair use permits transformative works. It has yet to be decided whether LLMs and their output qualify as transformative, or even if the training is capable of infringing copyright of an individual work in the first place if they're not reproducing it. In fact, there's a good amount of evidence which indicates that fair use _does_ apply, given how Google operates and what they've argued successfully (https://en.wikipedia.org/wiki/Perfect_10,_Inc._v._Amazon.com...).

Purchasing licenses when you are already entitled to your current use of the work is just bad business, especially when the legal precedent hasn't been set to know what rights might need to exist in said license.

You might not like the idea of your blog posts or other publicly posted materials being used to train LLMs, but that doesn't make it illegal (morality is subjective and I'm not about to argue one way or another). If it's really that much of a problem, you _do_ have the ability to remove your information from public accessibility, or otherwise protect it against LLM ingestion (IP restrictions, etc.).

edit: I am not a lawyer (this is likely obvious to any lawyers out there); this is my personal take.

Re: Perplexity AI is lying about their user agent

#436

Post by Dylan16807 »

denton-scratch wrote: post #423
lolinder wrote: post #286

Earlier quoted context omitted.

> It's in the title of TFA: they're being dishonest about who they are. PerplexityBot seems to understand that robots.txt is addressed to it. First, I'm ignoring the output of Perplexity. I have no reason to believe that they gave the LLM any knowledge about its internal operations, it's just riffing off of what OP is saying. Second, PerplexityBot is the user agent that they use when crawling and indexing. They never…

> (which are notably not the same as crawling) Is that a distinction without a difference? I think the robots.txt RFC was addressed specifically to crawlers ; so technically "ad hoc" requests generated automatically (i.e. by robots) aren't included. But the distinction operators would like to make is between humans and automata. Whether some automaton is a crawler or not isn't relevant.

If explicitly telling it to access a URL is an access by automaton, then isn't every web browser load an access by automaton?

Re: Perplexity AI is lying about their user agent

#437

Post by zzo38computer »

lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

If the user specifically asks for a file and asks a computer program to process it in a specific way, it should be permitted, regardless of user-agent spoofing (although user-agent spoofing should (normally) ideally only be done when the user specifically requests it; it should not do so automatically). However, this is better when using FOSS and/or local programs (or if the user is accessing them through a proxy, VPN, Tor, etc). Furthermore, any company that provides such services should not use unethical business practices, false advertising, etc, to do so.

If the company wants a copy of the files for your own use, then that is a bit different. When accessing large number of files at once, robots.txt is useful to block it. If they can get a copy of the files in a different way (assuming the files are intended to be public anyways), then they might do so. However, even in this case, still they should not use unethical business practices, false advertising, etc; and, they should also avoid user-agent spoofing.

(In this case, the reason for the user-agent spoofing does not seem to be deliberate, since it uses a headless browser. They should still change it though; probably by keeping the user-agent string but adding on a extra part such as "Perplexity", to indicate that it is what it is, in addition to the headless browser.)

Re: Perplexity AI is lying about their user agent

#438

Post by Dylan16807 »

blendergeek wrote: post #218
lolinder wrote: post #180

Earlier quoted context omitted.

I'm curious to know where you draw the line for what constitutes legitimate manipulation by a person and when it becomes distribution. I'm assuming that if I write code by hand for every part of the TCP/IP and HTTP stack I'm safe. What if I use libraries written by other people for the TCP/IP and HTTP part? What if I use a whole FOSS web browser? What about a paid local web browser? What if I run a script that I wrot…

Where exactly you crossed the line is a question for the courts. I am not a lawyer and will there for not help with the specifics. However, please see the Aereo case [0] for a possibly analogous case. I am allowed to have a DVR. There is no law preventing me from accessing my DVR over a network. Or possibly even colocating it in a local data center. But Aereo definitely crossed a line. Also see Vidangel [1]. The fact…

Which is offensive and the legal structure underlying that should be changed. Renting out machines, where a person could legally install and use the exact same machine, makes zero sense to count as "distribution".

Re: Perplexity AI is lying about their user agent

#439

Post by marcus0x62 »

stevenhuang wrote: post #428
marcus0x62 wrote: post #387

Earlier quoted context omitted.

The entire premise of the parent posters comment was that this is specifically unethical , so you lost me at the part where you deliberately decided to not address that in your reply.

> when we get in to the business of trying to force people to consume content in particular ways it's a bit icky in my opinion. ?

Read it again. Especially this part:

> Whether or not being an ass is unethical I'll leave to the philosophers.

Re: Perplexity AI is lying about their user agent

#440

Post by reissbaker »

gpm wrote: post #164
lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

> The second concern, though, is can perplexity do a live web query to my website and present data from my website in a format that the user asks for? Arguing that we should ban this moves into very dangerous territory. This feels like the fundamental core component of what copyright allows you to forbid. > Everything from ad blockers to reader mode to screen readers do exactly the same thing that Perplexity is doing…

I actually don't see the legal distinction here. A browser with an ad blocker is also:

1. Asking for a copy of your content

2. Manipulating the content

3. Redistributing the content to the end-user who requested it

Ditto for the LLM that has been asked by the end user to fetch your content and show it to them (possibly with a manipulation step e.g. summarization).

I don't think there's a legal, copyright distinction between doing that on a server vs doing that on a local machine. And, for example, if there were a difference: using a browser on a remote desktop would be illegal, or using curl on a machine you were SSHed into would be illegal. Also, an LLM running locally on your machine (doing the exact same thing) would be legal!

I understand that it's inconvenient and difficult to monetize content when an LLM is summarizing it, and hard to upsell other pages on a website to users when they aren't coming to your website and are instead accessing it through an LLM. But legally I think there's not an obvious distinction on copyright grounds, and if there were (other than a very fine-grained ban on specifically LLMs accessing websites, without any general principle behind it), it would catch up a lot of legitimate behavior in the dragnet.

I'd also point out that in the U.S., search engines have passed the "Fair Use" test of exemption from copyright — I think it would be very hard to make a distinction between what a search engine is doing (which is on a server!) and what an LLM is doing based on trying to say copyright distinguishes between server vs client architectures.

Post reply on HN