Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity AI is lying about their user agent

rknight.me

Post reply on HN
Print view View original

451–460 of 555 posts

Re: Perplexity AI is lying about their user agent

#451

Post by tarr11 »

lompad wrote: post #358
danlitt wrote: post #349

Earlier quoted context omitted.

Well, I guess what I mean is if the situation is as I describe in my previous comment, then anyone who did have the money to fight it would be a shoe-in. It's a much stronger case than, for example, the ongoing lawsuits by Matthew Butterick and others ( https://llmlitigation.com/ ).

Thanks for the link, that's fantastic to hear! I'm seriously sick of that whole "laundering copyright via AI"-grift - and the destruction of the creative industry is already pretty noticable. All the creatives who brought us all those wonderful masterworks with lots of thought and talent behind, they're all going bankrupt and getting fired right now. It's truly a tragedy - the loss of art is so much more serious than…

> destruction of the creative industry is already pretty noticable.

Can you explain what you mean by this? I’d be interested to know what jobs have been lost to AI (or if you are talking about something else)

Re: Perplexity AI is lying about their user agent

#452

Post by emporas »

cpursley wrote: post #444
mrweasel wrote: post #157

Earlier quoted context omitted.

So if I get access to the Perplexity AI source code (I borrow it from a friend), read all of it, and reproduce it at some level, then Perplexity will be:" sure, that's fine no harm, no IP theft, no copyright violation, because you read it so we're good"? No, they would sue me for everything I got, and then some. That's the weird thing about these companies, they are never afraid to use IP law to go after others, but…

Funny enough, their prompts leaked: https://www.reddit.com/r/perplexity_ai/s/kn6i20kMLH And I’ve built a perplexity clone in about a day - it’s not that hard: search -> scrape results -> parse results —> summarize results -> summarize aggregate results into single summary. I’m really not sure I even see their moat.

What have you used if i may ask? It seems very simple indeed. What search API is best?

Also there is a program called html2text to throw out the html formatting so as to use less tokens. Have you used this or something similar?

Re: Perplexity AI is lying about their user agent

#453

Post by zzo38computer »

EGreg wrote: post #448
zzo38computer wrote: post #437

Earlier quoted context omitted.

If the user specifically asks for a file and asks a computer program to process it in a specific way, it should be permitted, regardless of user-agent spoofing (although user-agent spoofing should (normally) ideally only be done when the user specifically requests it; it should not do so automatically). However, this is better when using FOSS and/or local programs (or if the user is accessing them through a proxy, VP…

A user-agent requests the file using your credentials, eg a cookie or public key signature. It is transforming the content for you , an authorized party. That is not the same as then making derivative copies and distributing the information to others without paying. For example, if I bought a ticket to a show, taped it and then distributed it to everyone, disregarding that the show prohibited this . If I shared my Ne…

Well, I am opposed to copyright. If it is publicly available, then you can make a copy, and even a modified version (as long as you do not claim that it is the same as the original).

However, what you say about credentials is still valid in the case of private data; this is why you should run the program locally and not use some other company's remote service for this use. (Well, it is one reason why. Other reason is all of the other bad stuff they do with the service.)

It is also valid about credentials, even if it is published but requires a password to access using that service; but even then, if you would ignore copyright, you can just use a different copy of the same file (which you might make by yourself).

None of this is meaning that you cannot pay for it, if they accept payment. It is also not meaning that whoever made it is required to give it away for free. What it is meaning, is that if you have a copy, you do not have to worry about copyright and other legal mess; you can just to do it; a license is not required.

However, it is also another issue how much power big companies are wasting with your data, whether they are authorized to access it or not. This is potentially a reason to disallow some uses, but that is independent from copyright (which is bad, anyways).

Re: Perplexity AI is lying about their user agent

#454

Post by zzo38computer »

insane_dreamer wrote: post #207
lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

> I don't want to live in a world where website owners can use DRM to force me to display their website in exactly the way that their designers envisioned it. I'm okay with this world, as a tradeoff. I'm not sure users should have _the right_ to reformat others' content.

Users should have the right to reformat their own copy of others content (automatically as well as manually). However, if they then redistribute the reformatted copy, then they should not be allowed to claim that it is the same formatting as the original, because it is not the same as the original.

Re: Perplexity AI is lying about their user agent

#455

Post by gpm »

reissbaker wrote: post #440
gpm wrote: post #164

Earlier quoted context omitted.

> The second concern, though, is can perplexity do a live web query to my website and present data from my website in a format that the user asks for? Arguing that we should ban this moves into very dangerous territory. This feels like the fundamental core component of what copyright allows you to forbid. > Everything from ad blockers to reader mode to screen readers do exactly the same thing that Perplexity is doing…

I actually don't see the legal distinction here. A browser with an ad blocker is also: 1. Asking for a copy of your content 2. Manipulating the content 3. Redistributing the content to the end-user who requested it Ditto for the LLM that has been asked by the end user to fetch your content and show it to them (possibly with a manipulation step e.g. summarization). I don't think there's a legal, copyright distinction…

The difference isn't so much the server, but the third party. You're allowed to modify computer programs (websites) as part of using them. You aren't allowed to then transfer the modified version (see section 117 of the US copyright code).

If you're in control of the server there's a plausible argument that you aren't transferring it. When perplexity is in control of the server... I don't see it. A traditional ad-blocker isn't "redistributing the content to the end-user who requested it" because it's the end user who has ownership over the data the whole time (note: not the copyright, the actual individual instance of the data). Unlike with a server run by a third party there is no third party legal entity who ever has the data.

You could conceivably make "ublock origin except it's a proxy run by a third party and we modify the website on the proxy", I'd agree that that has the same problem as a service like perplexity (though a different fair use analysis and I'm not sure what way that would go).

> I'd also point out that in the U.S., search engines have passed the "Fair Use" test of exemption from copyright — I think it would be very hard to make a distinction between what a search engine is doing (which is on a server!) and what an LLM is doing based on trying to say copyright distinguishes between server vs client architectures.

Well, sure. It's easy to distinguish between an LLM summarizing content and a traditional search engine though (and in ways relevant to the fair use analysis), just not based on the server client architecture.

Disclaimer: Not a lawyer, not legal advice, and so on.

Re: Perplexity AI is lying about their user agent

#456

Post by anileated »

putlake wrote: post #175
lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

It's funny I posted the inverse of this. As a web publisher, I am fine with folks using my content to train their models because this training does not directly steal any traffic. It's the "train an AI by reading all the books in the world" analogy. But what Perplexity is doing when they crawl my content in response to a user question is that they are decreasing the probability that this user would come to by content…

> I am fine with folks using my content to train their models because this training does not directly steal any traffic. It's the "train an AI by reading all the books in the world" analogy. But what Perplexity is doing when they crawl my content in response to a user question is that they are decreasing the probability that this user would come to by content (via Google, for example). This is unacceptable.

This appears to be self-contradictory. If you let an LLM to be trained* on “all the books” (posts, articles, etc.) in the world, the implication is that your potential readers will now simply ask that LLM. Not only will they pay Microsoft for that privilege, while you would get zilch, but you would not even know they ever read the fruits of your research.

* Incidentally, thinking of information acquisition by an ML model as if it was similar to human reading is a problematic fallacy.

Re: Perplexity AI is lying about their user agent

#457

Post by gpm »

SpaceNugget wrote: post #442
gpm wrote: post #279

Earlier quoted context omitted.

Great, so we agree that your previous comment asking I address "paid browsers" in particular was an unnecessary distraction. > I have a very hard time seeing an ethical distinction between running some proprietary code on my machine to complete a task and running some proprietary code on a cloud server to complete a task It's important to recognize that copyright is entirely artificial. Congress went "let's grant cre…

How is using perplexity any more so making a copy than your browser is making a copy? Unless you are distributing your website on thumb drives or floppy disks all distribution is achieved by making a copy. That's how networks work. Your logic would also imply that viewing a website through a VPN not operated by yourself would require the VPN operator to have a redistribution license for all the content on the website…

> How is using perplexity any more so making a copy than your browser is making a copy

Both are, the difference is that your browser doesn't transfer the copy to a new legal entity after modifying it. Rather the browser is under the control of the end user and the end user owns the data (not the copyright, but the actual instance of the data) the whole time.

> Your logic would also imply that viewing a website through a VPN not operated by yourself would require the VPN operator to have a redistribution license for all the content on the website which is not the case.

It doesn't because the VPN doesn't modify it, and the law explicitly distinguishes between the two cases and allows for transferring in the case of exact copies (provided you transfer all rights). I left this part of section 117 out because it wasn't relevant, but I'll quote it here

> Any exact copies prepared in accordance with the provisions of this section may be leased, sold, or otherwise transferred, along with the copy from which such copies were prepared, only as part of the lease, sale, or other transfer of all rights in the program. [And then the portion of the paragraph I quoted above] Adaptations so prepared may be transferred only with the authorization of the copyright owner.

> How do you think google is able to scrape whatever they like and redistribute summaries of the pages they have visited without consulting everyone who has ever made a website for a redistribution license.

A fair use argument, which I think is less likely (and I'd go so far as to say unlikely) to apply to a service like perplexity.ai but is ultimately a judgement call that will be made by the legal system and like all fair use arguments has no clear boundaries.

Re: Perplexity AI is lying about their user agent

#458

Post by anileated »

SpaghettiCthulu wrote: post #427
JumpCrisscross wrote: post #226

Earlier quoted context omitted.

> it's not scraping, it's retrieving the page on request from the user Search engines already tried it. It’s not retrieving on request because the user didn’t request the page, they requested a bot find specific content on any page.

That's not how it works in this case. The author asked the AI for information about a specific page.

The only way available to immediately test whether Perplexity pretends not to be Perplexity is by actively requesting a page. The fact that they mask their UA in that scenario makes it fairly obvious that they are not above bending rules and “working around” inconvenient for them public conventions. It seems safe to assume, until proven otherwise, that they would fake their bots’ user agents in every other case, such as when acquiring training data.

Re: Perplexity AI is lying about their user agent

#460

Post by 627467 »

I'm martian and I learned to use TCP/IP to make requests to IP addresses on Earth internet and interpret any response I get, however I'd like. I have been enjoying myself but recently came across some bruhaha around robot.txt, user agents and blah and apparently I'm not allowed to do whatever I want with the responses I get from my requests. I'm confused: you're willingly responding to my requests with strings of 0s and 1s but somehow you expect me to honor some arbitrary "convention" on what I can do with those 1s and 0s. earthlings are odd.
Post reply on HN