There is zero obligation for any client to present any particular user agent. If you don't want your content to be read, don't put it on the web.
Perplexity AI is lying about their user agent
331–340 of 555 posts
Re: Perplexity AI is lying about their user agent
#332There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…
What will happen if: Website owners decide to stop publishing because it’s not rewarded by a real human visit anymore? Then perplexity and the like won’t have new information to train their models on and no sites to answer the questions. I think there is a real content dilemma here at work. The incentives of Google and website owners were more or less aligned. This is not the case with perplexity.
> I think there is a real content dilemma here at work
It's not really a dilemma.
This is exactly what copyright serves to protect authors from. Perplexity copied the content, and in doing so directly competes with the original work, destroying it's market value and driving the original author out of business. Literally what copyright was invented to prevent.
It's the exact same situation as journalists going after Google & social media embeds of articles, which these sites propagandized as "prohibiting hyperlinking", but the issue has been the embedded (summary of the) content. Which people don't click through, and this is the entire point of those features for platforms like Facebook; Keeping users on facebook and not leaving.
This is why quite a few jurisdictions agreed with the journalists and moved to institute restrictions on such embedding.
By all practical considerations, perplexity is doing the exact same thing and trying to deflect with "we used an AI to paraphrase".
> The incentives of Google and website owners were more or less aligned.
The key difference here is that linking is and always has been fine. Google's Book search feature is fair use because the purpose is to send you to the book you searched for, not substitute the book.
Google's current AI summary feature is effectively the same as Perplexity. People don't click through to the original site, the original site doesn't get ad impressions or other revenue, and is driven out of business.
> What will happen if:
What will happen is what already is happening: Journalists are driven out of business, replaced by AI slop.
And then what? AI needs humans creating original content, especially for things like journalism and fact-finding. It'd be an eternal AI winter, all LLMs doomed to be stuck in 2025.
It's in every AI developer's best interest to halt the likes of Perplexity immediately before they irreparably damage the field of AI.
Re: Perplexity AI is lying about their user agent
#333Earlier quoted context omitted.
It's funny I posted the inverse of this. As a web publisher, I am fine with folks using my content to train their models because this training does not directly steal any traffic. It's the "train an AI by reading all the books in the world" analogy. But what Perplexity is doing when they crawl my content in response to a user question is that they are decreasing the probability that this user would come to by content…
> they are decreasing the probability that this user would come to by content (via Google, for example). Google has been providing summaries of stuff and hijacking traffic for ages. I kid you not, in the tourism sector this has been a HUGE issue, we have seen 50%+ decrease in views when they started doing it. We paid gazzilions to write quality content for tourists about the most different places just so Google could…
While I personally believe it should be opt-in, Google does have multiple ways to opt out of snippets while still being indexed. [1]
[1] https://developers.google.com/search/docs/appearance/snippet...
Re: Perplexity AI is lying about their user agent
#334Earlier quoted context omitted.
Right, it's ironic we spent 30 years fighting piracy and then suddenly corporations start doing it and now it's suddenly ok.
For me, the irony is the opposite side of the same coin, 30 years of "information wants to be free" and "copyright infringement isn't piracy" and "if you don't want to be indexed, use robots.txt"… …and then suddenly OpenAI are evil villains, and at least some of the people denounced them for copyright infringement are, in the same post, adamant that the solution is to force the model weights to become public domain.
I broadly agree with you, but I don't see what's contradictory about the solution of model weights becoming public domain.
When it comes to piracy, the people who have viewed it as ethical on the grounds that "information wants to be free" generally also drew the line at profiting from it: copying an MP3 and giving it to your friend or even a complete stranger is ethical, charging a fee for that (above and beyond what it costs you to make a copy) is not. From that perspective, what OpenAI is doing is evil not because they are infringing on everyone's copyright, but that they are profiting from it.
Re: Perplexity AI is lying about their user agent
#335If you've ever tried to do any web scraping, you'll know why they lie about the User-Agent, and you'd do it too if you wanted your program to work properly. Discriminating based on User-Agent string is the unethical part.
>and you'd do it too if you wanted your program to work properly. If I know the creator of the page doesn't want his page used by my program I wouldn't do it. >Discriminating based on User-Agent string is the unethical part. Not being exploited by an AI company is unethical? Robots.txt is made for telling bot identified by user agent what they are allowed to read.
> Robots.txt is made for telling bot identified by user agent what they are allowed to read.
Specifically it's meant for instructing "automatic clients known as crawlers" [0]. A crawler is defined by MDN as "a program, often called a bot or robot, which systematically browses the Web to collect data from webpages." [1]
As generally understood, wget is not a crawler even though it may be used to build one. Neither is curl. A crawler is a program which systematically browses the web, usually to build a search index.
I see no evidence that Perplexity's crawler is ignoring robots.txt, I only see evidence that when a user does a one-off request for a specific URL then Perplexity uses Chrome to access the site.
Basically, OP is using the wrong tool for the job and complaining when it doesn't work. If he wants to be excluded from Perplexity for one-off requests (as distinct from crawling) he needs to reach out to them, there is no applicable RFC.
[0] https://www.rfc-editor.org/rfc/rfc9309.html
[1] https://developer.mozilla.org/en-US/docs/Glossary/Crawler
Re: Perplexity AI is lying about their user agent
#336Earlier quoted context omitted.
Can’t wait for OpenAI to settle with The New York Times. For a billion dollars no less.
I, on the other hand, hope NYT refuses a settlement and OpenAI loses in court.
Be careful what you wish for, because, depending on how broad the reasoning in such a decision would be, it is not impossible that the precedent would be used to then target ad blockers and similar software.
Re: Perplexity AI is lying about their user agent
#337Earlier quoted context omitted.
Main issues: 1) Schools use primarily public domain knowledge for education. It's rarely your private blog post being used to mostly learn writing blog posts. 2) There's no attribution, no credit. Public academia is heavily based (at least theoretically) on acknowledging every single paper you built your thesis on. 3) There's no payment. In school (whatever level) somebody's usually paying somebody for having worked…
1) If your blog posts are private, why are they on publicly accessible websites? Why not put it behind a paywall of some sort? 2) How many novels have bibliographies? How many musicians cite their influences? Citing sources is all well and good in academic papers, but there’s a point at which it just becomes infeasible. The more transformative the work, the harder it is to cite inspiration. 3) What about libraries? S…
> 1) If your blog posts are private, why are they on publicly accessible websites? Why not put it behind a paywall of some sort?
If I grow apple trees in front of my house and you come and take all apples and then turn up at my doorstep trying to sell me apple juice made from the apples you nicked that doesn't mean you had the right to do it, because I chose not to build a tall fence around my apple trees. Public content is free to read for humans, not free for corporations to offer paid content generation services based on my public content taken without me knowing or being asked for permission.
> 2) How many novels have bibliographies? How many musicians cite their influences? Citing sources is all well and good in academic papers, but there’s a point at which it just becomes infeasible. The more transformative the work, the harder it is to cite inspiration.
You are making this kind of argument: "How much is a drop of gas? Nothing. Right, could you fill my car drop by drop?"
If we have technology that can charge for producing bullshit on an industrial scale by recombining sampled works of others, we are perfectly capable of keeping track of the sources used for training and generative diarrhoea.
> 3) What about libraries? Should they be licensing every book they have in their collections? Should the people who check the books out have to pay royalties to learn from them?
Re: Perplexity AI is lying about their user agent
#338Earlier quoted context omitted.
Or, I return whatever content I want, within the bounds of the law, based on whatever parameters I decide. What's your problem with that? Again, connect to my server or don't. But don't tell me what type of response I'm obligated to provide you. If I think a given request is from an LLM training module, I don't have any legal obligation whatsoever to return my original content. Or a 400-series response. If I want to…
But nobody is arguing for that. Instead, what the server owners want is to mandate the clients connecting to them to provide enough information to reliably reject such connections.
There are literally people in this thread arguing that it is "unethical" to discriminate based on user agent.
Re: Perplexity AI is lying about their user agent
#339Earlier quoted context omitted.
What will happen if: Website owners decide to stop publishing because it’s not rewarded by a real human visit anymore? Then perplexity and the like won’t have new information to train their models on and no sites to answer the questions. I think there is a real content dilemma here at work. The incentives of Google and website owners were more or less aligned. This is not the case with perplexity.
How would an LLM training on your writing reduce your reward? I guess if you're doing it for a living sure, but most content I consume online is created without incentive (social media, blogs, stack overflow). I write a fair amount and have been for a few years. I like to play with ideas. If an llm learned from my writing and it helped me propagate my ideas, I'd be happy. I lose on social status imaginary internet po…
> I guess they do it for the glory of having their handle next to the answer?
Yes, it's hardly surprising that people find upvotes and direct social rewards more exciting than being slurped somewhere into GPT-4's weights.
Re: Perplexity AI is lying about their user agent
#340There is zero obligation for any client to present any particular user agent. If you don't want your content to be read, don't put it on the web.
Reading is completely fine as this is author's intention.
Using someone else's content in commercial purposes for free is absolutely not -- are you saying that we should ignore copyrights and all that since something is on the web?
If I, as ordinary person, wanted to do that to a company, that company would call me a thief. So I think it's only fair to apply same logic to them.