Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity AI is lying about their user agent

rknight.me

Post reply on HN
Print view View original

301–310 of 555 posts

Re: Perplexity AI is lying about their user agent

#301

Post by SpaghettiCthulu »

putlake wrote: post #175
lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

It's funny I posted the inverse of this. As a web publisher, I am fine with folks using my content to train their models because this training does not directly steal any traffic. It's the "train an AI by reading all the books in the world" analogy. But what Perplexity is doing when they crawl my content in response to a user question is that they are decreasing the probability that this user would come to by content…

You're missing the part where Perplexity still makes a request each time it's asked about the URL. You still get the traffic!

Re: Perplexity AI is lying about their user agent

#302

Post by mattigames »

bluish29 wrote: post #287
mattigames wrote: post #270

Earlier quoted context omitted.

Its fairly logical to assume that robots.txt governs robots (empahsis in "bots") not just crawlers, if they are only intended to block crawlers why aren't they called crawlers.txt instead and remove all ambiguity?

That's a historical question. At this time, most if not all the bots were either search engines or archival. The name was even "RobotsNotWanted.txt" at the beginning but made "robots.txt" for simplicity. To give another example, Internet Archive stopped respecting it a couple of years ago, and they discuss this point (crawlers vs other bots) here [1]. [1] https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea..…

If it was uniquely an historical question then another text file to handle AI requests would exist by now, e.g. ai-bots.txt, but it hasn't and likely never will, they don't want to even have to pretend to comply with creator requests about forbidding (or not) the usage of their sites.

Re: Perplexity AI is lying about their user agent

#303

Post by treyd »

lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

Yeah if people get to extensive about blocking then we're going to end up with a scenario where the web request functionality is implementing by telling the chatbot's users's browser to make the fetch and submit it back to the server for processing, making it largely indistinguishable from the user making the query themselves. If CORS gets in the way they can just prompt users to install a browser extension to use the web request functionality.

Re: Perplexity AI is lying about their user agent

#304

Post by alexey-salmin »

JumpCrisscross wrote: post #226
ulrikrasmussen wrote: post #179

Earlier quoted context omitted.

But it's not scraping, it's retrieving the page on request from the user.

> it's not scraping, it's retrieving the page on request from the user Search engines already tried it. It’s not retrieving on request because the user didn’t request the page, they requested a bot find specific content on any page.

But it's not what happened here. It WAS retrieving on request.

> I went into Perplexity and asked "What's on this page rknight.me/PerplexityBot?". Immediately I could see the log and just like Lewis, the user agent didn't include their custom user agent

Re: Perplexity AI is lying about their user agent

#305

Post by skilled »

insane_dreamer wrote: post #238
skilled wrote: post #28

Earlier quoted context omitted.

Can’t wait for OpenAI to settle with The New York Times. For a billion dollars no less.

I, on the other hand, hope NYT refuses a settlement and OpenAI loses in court.

Same, for sure!

Re: Perplexity AI is lying about their user agent

#306

Post by nradov »

baxtr wrote: post #183
lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

What will happen if: Website owners decide to stop publishing because it’s not rewarded by a real human visit anymore? Then perplexity and the like won’t have new information to train their models on and no sites to answer the questions. I think there is a real content dilemma here at work. The incentives of Google and website owners were more or less aligned. This is not the case with perplexity.

A lot of the public website content targeted towards consumers is already SEO slop trying to sell you something or maximize ad revenue. If those website owners decide to stop publishing due to lack of real human visits then little of value will be lost. Much of the content with real value for consumers has already moved to sites that require registration (and sometimes payment) for access.

For technical content of value to professionals, much of that is hosted by vendors or industry organizations. Those tend to get their revenue in other ways and don't care about companies scraping their content for AI model training. Like the IETF isn't going to stop publishing new RFCs just because Perplexity uses them.

Re: Perplexity AI is lying about their user agent

#307

Post by mattigames »

lolinder wrote: post #296
gcanyon wrote: post #282

Earlier quoted context omitted.

It seems self-evident to me that if a user tells a bot to go get a web page, robots.txt doesn't apply, and the bot shouldn't respect it. I understand others' concerns that, like Apple's reader, and other similar tools, it's ethically debatable whether a site should be required to comply with the request, and spoofing an agent seems in dubious territory. I don't think a good answer has been proposed for this challenge…

> spoofing an agent seems in dubious territory. Just to clarify, Perplexity is not spoofing a user agent, they're legitimately using a headless Chrome to fetch the page. The author just misunderstood their docs [0]: when they say that "you can identify our web crawler by its user agent", they're talking about the crawler, not the browser they use for ad hoc requests. As you note, crawling is different. [0] https://do…

This is completely false, the user agent being used by Perplexity its _not_ the headless-chrome user agent, wich is close similar to this (emphasis on HeadlessChrome):

    Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/119.0.0.0 Safari/537.36
They are spoofing it to pretend to be a desktop Chrome one:

    Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36

Re: Perplexity AI is lying about their user agent

#308

Post by mftrhu »

dr_dshiv wrote: post #198
hiatus wrote: post #118

Earlier quoted context omitted.

Computers are not people. Laws differ and consequences can be different based on the actor (like how minors are treated differently in courts). Just because a person can do it does not automatically mean those same rights transfer to arbitrary machines.

Corporations are people. Not saying that’s right. But is that not the law?

Corporations are legal persons, which are not the same as natural persons (AKA plain old human beings).

The law endows natural persons with many rights which cannot and do not apply to legal persons - corporations, governments, cooperatives and the like can enter into contracts (but not marriage contracts), own property (which will not be protected by things like homestead laws and the such), sue, and be sued. They cannot vote, claim disability exemptions, or have any rights to healthcare and the like, while natural persons do.

Legal persons are not treated and do not have to be treated like natural persons.

Re: Perplexity AI is lying about their user agent

#309

Post by deely3 »

esha_manideep wrote: post #107
bastawhiz wrote: post #53

I have a silly website that just proxies GitHub and scrambles the text. It runs on CF Workers. https://guthib.mattbasta.workers.dev For the past month or two, it's been hitting the free request limit as some AI company has scraped it to hell. I'm not inclined to stop them. Go ahead, poison your index with literal garbage. It's the cost of not actually checking the data you're indiscriminately scraping.

They check after they scrape

How? Real people read all millions of pages of internet texts to verify it?

Re: Perplexity AI is lying about their user agent

#310

Post by nradov »

marcus0x62 wrote: post #299
nradov wrote: post #278

Earlier quoted context omitted.

I'm having difficulty grasping the concept. Only a fool would trust any HTTP headers such as User-Agent sent by a random unauthenticated client. Your expenses are your problem.

… and I have absolutely no obligation to provide any particular response to any particular client. Parsing, rendering, and trusting that the payload is consistent from request to request is your problem . You can connect to my server, or not. I really don’t care. What you cannot do is dictate how my server responds to your request.

Sure. So just return an HTTP 4XX response to requests you don't like. What's the problem?
Post reply on HN