Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity AI is lying about their user agent

rknight.me

Post reply on HN
Print view View original

311–320 of 555 posts

Re: Perplexity AI is lying about their user agent

#311

Post by wtf242 »

The amount of AI bots scraping/indexing content is just mind boggling. for my books site https://thegreatestbooks.org, without blocking any bots, I was probably getting 500,000~ requests a day from ONLY ai bots. Claudebot, amazon ai bot, bing ai bot, bytespider, openai. Endless ai bots just non-stop indexing/scraping my data.

Before i moved my dns to cloudflare and got on their pro plan, which offers robust bot blocking, they were severely hurting my performance to the point that I bought a new server to offload the traffic.

Re: Perplexity AI is lying about their user agent

#312

Post by SamBam »

JumpCrisscross wrote: post #226
ulrikrasmussen wrote: post #179

Earlier quoted context omitted.

But it's not scraping, it's retrieving the page on request from the user.

> it's not scraping, it's retrieving the page on request from the user Search engines already tried it. It’s not retrieving on request because the user didn’t request the page, they requested a bot find specific content on any page.

This is why this conversation is making me insane. How are people saying straight-faced that the user is requesting a specific page? They aren't, they're doing a search of the web.

That's not at all the same as a browser visiting a page.

Re: Perplexity AI is lying about their user agent

#313

Post by lolinder »

mattigames wrote: post #307
lolinder wrote: post #296

Earlier quoted context omitted.

> spoofing an agent seems in dubious territory. Just to clarify, Perplexity is not spoofing a user agent, they're legitimately using a headless Chrome to fetch the page. The author just misunderstood their docs [0]: when they say that "you can identify our web crawler by its user agent", they're talking about the crawler, not the browser they use for ad hoc requests. As you note, crawling is different. [0] https://do…

This is completely false, the user agent being used by Perplexity its _not_ the headless-chrome user agent, wich is close similar to this (emphasis on HeadlessChrome): Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/119.0.0.0 Safari/537.36 They are spoofing it to pretend to be a desktop Chrome one: Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.…

Ah, you're correct, my bad.

I don't personally have a problem with spoofing user agents, but yeah, they're either spoofing or for some reason they're truly using a non-headless Chrome.

Re: Perplexity AI is lying about their user agent

#314

Post by epolanski »

CobrastanJorji wrote: post #261
epolanski wrote: post #188

Earlier quoted context omitted.

> they are decreasing the probability that this user would come to by content (via Google, for example). Google has been providing summaries of stuff and hijacking traffic for ages. I kid you not, in the tourism sector this has been a HUGE issue, we have seen 50%+ decrease in views when they started doing it. We paid gazzilions to write quality content for tourists about the most different places just so Google could…

I'm curious about the tourism sector problem. In tourism, I would think the goal would be to promote a location. You want people to be able to easily discover the location, get information about it, and presumably arrange to travel to those locations. If Google gets the information to the users, but doesn't send the tourist to the website, is that harmful? Is it a problem of ads on the tourism website? Or is more of…

We would employ local guides all around the world to craft itinerary plans to visit places, give tips, tricks, recommend experiences and places (we made money by selling some of those through our website) and it was a success.

Customers liked the in depth value of that content and it converted to buys (we sold experiences and other stuff, sort of like getyourguide).

One day all of our content ended up on Google "what time is best to visit the Sagrada Familia" and you would have a copy pasted answer by Google.

This killed a lot of traffic.

Anyway, I just wanted to point out that the previous user was a bit naive taking his fight to LLMs when search engines and OSs have been leeching and hijacking content for ages.

Re: Perplexity AI is lying about their user agent

#315

Post by danlitt »

putlake wrote: post #175
lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

It's funny I posted the inverse of this. As a web publisher, I am fine with folks using my content to train their models because this training does not directly steal any traffic. It's the "train an AI by reading all the books in the world" analogy. But what Perplexity is doing when they crawl my content in response to a user question is that they are decreasing the probability that this user would come to by content…

I'm not sure what you mean exactly. If Perplexity is actually doing something with your article in-band (e.g. downloading it, processing it, and present that processed article to the user) then they're just breaking the law.

I've never used that tool (and don't plan to) so I don't know. If they just embed the content in an iframe or something then there's no issue (but then there's no need or point in scraping). If they're just scraping to train then I think you also imply there's no issue. If they're just copying your content (even if the prompt is "Hey Perplexity, summarise this article ") then that's vanilla infringement, whether they lie about their UA or not.

Re: Perplexity AI is lying about their user agent

#316

Post by dragonwriter »

tomxor wrote: post #170

> Not sure where we go from here. I don't want my posts slurped up by AI companies for free[1] but what else can I do? You can sprinkle invisible prompt injections throughout your content to override the user's prompts and control the LLM's responses. Rather than alerting the user that it's not allowed, you make it produce something plausible but incorrect i.e silently deny access, to avoid counter prompts, so it's h…

> Also weirdly it cannot spell behaviour.

https://www.merriam-webster.com/dictionary/behavior

Re: Perplexity AI is lying about their user agent

#317

Post by lolinder »

SamBam wrote: post #312
JumpCrisscross wrote: post #226

Earlier quoted context omitted.

> it's not scraping, it's retrieving the page on request from the user Search engines already tried it. It’s not retrieving on request because the user didn’t request the page, they requested a bot find specific content on any page.

This is why this conversation is making me insane. How are people saying straight-faced that the user is requesting a specific page? They aren't, they're doing a search of the web. That's not at all the same as a browser visiting a page.

Because that's literally what the author does in TFA and then complains about when Perplexity complies.

> What is this post about https://rknight.me/blog/blocking-bots-with-nginx/

Re: Perplexity AI is lying about their user agent

#318

Post by ADeerAppeared »

bko wrote: post #211
baxtr wrote: post #183

Earlier quoted context omitted.

What will happen if: Website owners decide to stop publishing because it’s not rewarded by a real human visit anymore? Then perplexity and the like won’t have new information to train their models on and no sites to answer the questions. I think there is a real content dilemma here at work. The incentives of Google and website owners were more or less aligned. This is not the case with perplexity.

How would an LLM training on your writing reduce your reward? I guess if you're doing it for a living sure, but most content I consume online is created without incentive (social media, blogs, stack overflow). I write a fair amount and have been for a few years. I like to play with ideas. If an llm learned from my writing and it helped me propagate my ideas, I'd be happy. I lose on social status imaginary internet po…

> How would an LLM training on your writing reduce your reward?

Because you're not getting the ad impressions anymore. The harsh reality is that people do not click on to sources, so when sites like Perplexity copy your content, you lose the revenue on that content.

This, in turn, drives all real journalism out of business. And then everyone's screwed, including these AI reposting sites.

Re: Perplexity AI is lying about their user agent

#320

Post by marcus0x62 »

nradov wrote: post #310
marcus0x62 wrote: post #299

Earlier quoted context omitted.

… and I have absolutely no obligation to provide any particular response to any particular client. Parsing, rendering, and trusting that the payload is consistent from request to request is your problem . You can connect to my server, or not. I really don’t care. What you cannot do is dictate how my server responds to your request.

Sure. So just return an HTTP 4XX response to requests you don't like. What's the problem?

Or, I return whatever content I want, within the bounds of the law, based on whatever parameters I decide. What's your problem with that? Again, connect to my server or don't. But don't tell me what type of response I'm obligated to provide you.

If I think a given request is from an LLM training module, I don't have any legal obligation whatsoever to return my original content. Or a 400-series response. If I want to intersperse a paragraph from Don Quixote between every second sentence, that's my call.

Post reply on HN