If the cause of training LLMs is so noble then surely an opt in model would work, no?
Perplexity AI is lying about their user agent
361–370 of 555 posts
Re: Perplexity AI is lying about their user agent
#362> Next up is some kind of GDPR request perhaps? GDPR doesn't preclude anyone from scraping you. In fact, scraping is not illegal in any context (LinkedIn keeps losing lawsuits). Using copyrighted data in training LLMs is a huge grey area, but probably not outright illegal and will take like a decade (if not more) before we'll have legislative clarity.
Using copyrighted data in training LLMs is allowed in the European Union, unless the copyright holder specifically opts out. This is in the recent Artificial Intelligence Act, which defines AI training as a type of "data mining" being covered by the EU Directive 2019/790 Article 4.
The problem is that there is no designated protocol for opting out. There are a bunch of protocols pushed by different entities, and support is fragmented even where there is intent to do the right thing. This means of course that they don't work in practice.
An example: The most well known out-out protocol might be DeviantArt's "noai" and "noimageai" tags that could be in HTTP and/or HTML headers [1]. The web site Cara.app has got a large influx of artists recently because of its anti-AI stance. Cara.app puts only a "noai" metadata tag in HTML headers of pages that link to images but not in any HTTP response headers. Spawning.ai's "datadiligence" library for web crawlers [2] searchers for "noai" tags in HTTP response headers of image files but not in HTML files that link to them.
1. "noai" tag: https://www.deviantart.com/team/journal/UPDATE-All-Deviation...
2. "Datadiligence": https://github.com/Spawning-Inc/datadiligence/tree/main
Re: Perplexity AI is lying about their user agent
#363Earlier quoted context omitted.
Why is that the line and not a paid web browser? What about a paid web browser whose primary feature is a really powerful ad blocker?
Why are you ignoring his main argument?
I'm not. I'm asking why this flow is "distribution":
* User types an address into Perplexity
* Perplexity fetches the page, transforms it, and renders some part of it for the user
But this flow is not:
* User types an address into Orion Browser
* Orion Browser fetches the page, transforms it, and renders some part of it for the user
Regardless of the legal question (which I'm also skeptical of), I'm especially unconvinced that there's a moral distinction between a web service that transforms copyrighted works in an ad hoc manner upon a user's specific request and renders them for that specific user vs an installed application that does exactly the same thing.
Re: Perplexity AI is lying about their user agent
#364Earlier quoted context omitted.
It's funny I posted the inverse of this. As a web publisher, I am fine with folks using my content to train their models because this training does not directly steal any traffic. It's the "train an AI by reading all the books in the world" analogy. But what Perplexity is doing when they crawl my content in response to a user question is that they are decreasing the probability that this user would come to by content…
> they are decreasing the probability that this user would come to by content (via Google, for example). Google has been providing summaries of stuff and hijacking traffic for ages. I kid you not, in the tourism sector this has been a HUGE issue, we have seen 50%+ decrease in views when they started doing it. We paid gazzilions to write quality content for tourists about the most different places just so Google could…
> Google has been providing summaries of stuff and hijacking traffic for ages.
Yes, Google hijacked images for some time. But in general there has "always" been the option to tell Google not to display summaries etc with these meta tags:
Re: Perplexity AI is lying about their user agent
#365Earlier quoted context omitted.
Right, it's ironic we spent 30 years fighting piracy and then suddenly corporations start doing it and now it's suddenly ok.
For me, the irony is the opposite side of the same coin, 30 years of "information wants to be free" and "copyright infringement isn't piracy" and "if you don't want to be indexed, use robots.txt"… …and then suddenly OpenAI are evil villains, and at least some of the people denounced them for copyright infringement are, in the same post, adamant that the solution is to force the model weights to become public domain.
Evil villains to individuals, if what they were doing was actually open.
Then sure, but they're getting a pass because of capitalism and dcma was getting that same pass.
Re: Perplexity AI is lying about their user agent
#366Just the other day Perplexity CEO Aravind Srinivas was dunking on Google and OpenAI, and putting themselves on a superior moral position because they give citations while closed-book LLMs memorize the web information with large models and don't give credit. Funny they got caught not following robots.txt and hiding their identity. https://x.com/tsarnick/status/1801714601404547267
Nobody follows robots.txt, because every site's robots.txt forbids anybody that isn't google from looking at it. Also, "hiding their identity" is what every single browser does since Mosaic changed its name.
Including extra, legacy agents isn't hiding because they include their distinct identifiers too.
Re: Perplexity AI is lying about their user agent
#367Earlier quoted context omitted.
> it's not scraping, it's retrieving the page on request from the user Search engines already tried it. It’s not retrieving on request because the user didn’t request the page, they requested a bot find specific content on any page.
But it's not what happened here. It WAS retrieving on request. > I went into Perplexity and asked "What's on this page rknight.me/PerplexityBot?". Immediately I could see the log and just like Lewis, the user agent didn't include their custom user agent
That was to test the user-agent hiding. The broader problem—Perplexity laundering attribution—is where the scraping vs retrieval question comes into play.
Re: Perplexity AI is lying about their user agent
#368I have a silly website that just proxies GitHub and scrambles the text. It runs on CF Workers. https://guthib.mattbasta.workers.dev For the past month or two, it's been hitting the free request limit as some AI company has scraped it to hell. I'm not inclined to stop them. Go ahead, poison your index with literal garbage. It's the cost of not actually checking the data you're indiscriminately scraping.
[deleted]
Re: Perplexity AI is lying about their user agent
#369Earlier quoted context omitted.
What is a "visit"? TFA demonstrates that they got a hit on their site, that's how they got the logs. Is it necessary to load the JavaScript for it to count as a visit? What if I access the site with noscript? Or is it only a visit if I see all your recommended content? I usually block those recommendations so that I don't get distracted from the article I actually came to read—is my visit a less legitimate visit than…
A visit is a human reader. At the very least they get exposed to your website name. Notice your product/service if you get lucky. Become a customer at a later visit. We are talking about cutting the first step off so that everything which may come afterwards is cut off as well.
In other words, content is bait, reward is a captured user whose attention - whose sanity, the finite amount of life - can be wasted or plain used against them.
I'm more than happy to see all the websites with attention economy business models to shut down. Yes, that might be 90% of the Internet. That would be the 90% that is poisonous shit.
Re: Perplexity AI is lying about their user agent
#370The author has misunderstood when the perplexity user agent applies. Web site owners shouldn’t dictate what browser users can access their site with - whether that’s chrome, firefox, or something totally different like perplexity. When retrieving a web page _for the user_ it’s appropriate to use a UA string that looks like a browser client. If perplexity is collecting training data in bulk without using their UA that…
It’s not retrieving a web page though is it? It’s retrieving the content then manipulating it. Perplexity isn’t a web browser.
Yes, that's literally why "user agent" is called "user agent". It's a program that acts in place and in the interest of its user, and this in particular always included allowing the user to choose what will or won't be rendered, and how. It's not up to the server what the client does with the response they get.