[deleted]
Perplexity AI is lying about their user agent
281–290 of 555 posts
Re: Perplexity AI is lying about their user agent
#282There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…
It seems self-evident to me that if a user tells a bot to go get a web page, robots.txt doesn't apply, and the bot shouldn't respect it. I understand others' concerns that, like Apple's reader, and other similar tools, it's ethically debatable whether a site should be required to comply with the request, and spoofing an agent seems in dubious territory. I don't think a good answer has been proposed for this challenge, unfortunately.
Re: Perplexity AI is lying about their user agent
#283Earlier quoted context omitted.
Orion browser presuming it does what does what it's name says it does doesn't redistribute anything... so presumably not.
I asked you this in the other subthread, but what exactly is the moral distinction (I'm not especially interested in the legal one here because our copyright law is horribly broken) between these two scenarios? * User asks proprietary web browser to fetch content and render it a specific way, which it does * User asks proprietary web service to fetch content and render it a specific way, which it does The technical d…
Just put a long comment on the other thread addressing this.
Re: Perplexity AI is lying about their user agent
#284[deleted]
Re: Perplexity AI is lying about their user agent
#285Earlier quoted context omitted.
Right, it's ironic we spent 30 years fighting piracy and then suddenly corporations start doing it and now it's suddenly ok.
For me, the irony is the opposite side of the same coin, 30 years of "information wants to be free" and "copyright infringement isn't piracy" and "if you don't want to be indexed, use robots.txt"… …and then suddenly OpenAI are evil villains, and at least some of the people denounced them for copyright infringement are, in the same post, adamant that the solution is to force the model weights to become public domain.
The deal of the internet has always been: send me what you want and I’ll render it however I want. This includes feeding it into AI bots now. I don’t love being on the same side as these “AI” snakeoil salesmen, but they are following the rules of the road.
Robots.txt is just a voluntary thing. We’re going to see more and more of the internet shut off by technical means instead, which is a bummer. But on the bright side it might kill off the ad based model. Silver linings and all that.
Re: Perplexity AI is lying about their user agent
#286Earlier quoted context omitted.
What is a "visit"? TFA demonstrates that they got a hit on their site, that's how they got the logs. Is it necessary to load the JavaScript for it to count as a visit? What if I access the site with noscript? Or is it only a visit if I see all your recommended content? I usually block those recommendations so that I don't get distracted from the article I actually came to read—is my visit a less legitimate visit than…
> What exactly is Perplexity doing here that isn't okay that people don't already do with their local user agents? It's in the title of TFA: they're being dishonest about who they are. PerplexityBot seems to understand that robots.txt is addressed to it . It's understood that site operators have a right to use the User-Agent to discriminate among visitors; that's why robots.txt is a standard. Crawlers that disrespect…
> It's in the title of TFA: they're being dishonest about who they are. PerplexityBot seems to understand that robots.txt is addressed to it.
First, I'm ignoring the output of Perplexity. I have no reason to believe that they gave the LLM any knowledge about its internal operations, it's just riffing off of what OP is saying.
Second, PerplexityBot is the user agent that they use when crawling and indexing. They never claimed to use that user agent for ad hoc HTTP requests (which are notably not the same as crawling).
Third, I disagree that anyone has an obligation to be honest in their User-Agent. Have you ever looked at Chrome's user agent? They're spoofing just about everyone, as is every browser. Crawlers should respect robots.txt, but I'd be totally content if we just got rid of the User-Agent string entirely.
Re: Perplexity AI is lying about their user agent
#287Earlier quoted context omitted.
Just to go a little bit more into detail on this, because the article and most of the conversation here is based on a big misunderstanding: robots.txt governs crawlers . Fetching a single user-specified URL is not crawling. Crawling is when you automatically follow links to continue fetching subsequent pages. Perplexity’s documentation that the article links to describes how their crawler works. That is not the piece…
Its fairly logical to assume that robots.txt governs robots (empahsis in "bots") not just crawlers, if they are only intended to block crawlers why aren't they called crawlers.txt instead and remove all ambiguity?
That's a historical question. At this time, most if not all the bots were either search engines or archival. The name was even "RobotsNotWanted.txt" at the beginning but made "robots.txt" for simplicity. To give another example, Internet Archive stopped respecting it a couple of years ago, and they discuss this point (crawlers vs other bots) here [1].
[1] https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
Re: Perplexity AI is lying about their user agent
#288Earlier quoted context omitted.
> If the "content" had no value, why would google go through the effort of scraping it and presenting it to the user? They don't present it all, they summarize it. And let's be serious here, I was being polite because I don't know the OPs business. But 99% of this sort of content is SEO trash and contributes to the wasteland that the internet is becoming. Feel free to point me to the good stuff.
I would also think that the intrinsic value is different. If there is a hotel on a mountain writing "quality content" about the place, to them it really doesn't matter who "steals" their content, the value is in people going to the hotel on the mountain not in people reading about the hotel on the mountain. Like to society the value is in the hotel, everything else is just fluff around it that never had any real valu…
>Travel bloggers and vloggers
I've no doubt some good ones exist, but my instinct is to ignore every word this industry says because it's paid placement and our world is run by advertisers.
Re: Perplexity AI is lying about their user agent
#289Re: Perplexity AI is lying about their user agent
#290Well, one solution to this would be to include bulk Markov chain generated content on your website. I'm starting to think the only way to fight back against AI scraping, is to make ourselves as unappealing a target as possible. If you get 100 poisoned articles for every 1 good article, you become a waste of resources to scrape.
Simply use a Google Noindex directory on the pages you're using as an attack vector so they don't pollute your website's footprint.