Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity AI is lying about their user agent

rknight.me

Post reply on HN
Print view View original

291–300 of 555 posts

Re: Perplexity AI is lying about their user agent

#291

Post by SamBam »

CobrastanJorji wrote: post #261
epolanski wrote: post #188

Earlier quoted context omitted.

> they are decreasing the probability that this user would come to by content (via Google, for example). Google has been providing summaries of stuff and hijacking traffic for ages. I kid you not, in the tourism sector this has been a HUGE issue, we have seen 50%+ decrease in views when they started doing it. We paid gazzilions to write quality content for tourists about the most different places just so Google could…

I'm curious about the tourism sector problem. In tourism, I would think the goal would be to promote a location. You want people to be able to easily discover the location, get information about it, and presumably arrange to travel to those locations. If Google gets the information to the users, but doesn't send the tourist to the website, is that harmful? Is it a problem of ads on the tourism website? Or is more of…

Presumably the issue is more the travel guides/Time Out/Tripadvisor type websites.

They make money by you reading their stuff, not by you actually spending money in the place.

Re: Perplexity AI is lying about their user agent

#292

Post by gcanyon »

luaKmua wrote: post #245
itsoktocry wrote: post #232

Earlier quoted context omitted.

> We paid gazzilions to write quality content for tourists about the most different places just so Google could put it on their homepage. It's just depressing It's a legitimate complaint, and it sucks for your business. But I think this demonstrates that the sort of quality content you were producing doesn't actually have much value.

That line of thinking makes no sense. If the "content" had no value, why would google go through the effort of scraping it and presenting it to the user?

It's not that it has no value, it's that there is no established way (other than ad revenue) to charge users for that content. The fact that google is able to monetize ad revenue at least as well as, and probably better than, almost any other entity on the internet, means that big-G is perfectly positioned to cut out the creator -- until the content goes stale, anyway.

Re: Perplexity AI is lying about their user agent

#294

Post by bshacklett »

araes wrote: post #115
bmicraft wrote: post #70

Earlier quoted context omitted.

I think that's called a school

Main issues: 1) Schools use primarily public domain knowledge for education. It's rarely your private blog post being used to mostly learn writing blog posts. 2) There's no attribution, no credit. Public academia is heavily based (at least theoretically) on acknowledging every single paper you built your thesis on. 3) There's no payment. In school (whatever level) somebody's usually paying somebody for having worked…

1) If your blog posts are private, why are they on publicly accessible websites? Why not put it behind a paywall of some sort?

2) How many novels have bibliographies? How many musicians cite their influences? Citing sources is all well and good in academic papers, but there’s a point at which it just becomes infeasible. The more transformative the work, the harder it is to cite inspiration.

3) What about libraries? Should they be licensing every book they have in their collections? Should the people who check the books out have to pay royalties to learn from them?

Re: Perplexity AI is lying about their user agent

#295

Post by bpm140 »

With all the ad blockers out there, which functionally demonetize content sites, why isn’t there an ad equivalent to robots.txt that says “don’t display this site if ads are blocked”?

So many good comments from several points of view in this thread and the thing I can’t square is the same person championing ad blockers and condemning agents like Perplexity.

Re: Perplexity AI is lying about their user agent

#296

Post by lolinder »

gcanyon wrote: post #282
lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

It seems self-evident to me that if a user tells a bot to go get a web page, robots.txt doesn't apply, and the bot shouldn't respect it. I understand others' concerns that, like Apple's reader, and other similar tools, it's ethically debatable whether a site should be required to comply with the request, and spoofing an agent seems in dubious territory. I don't think a good answer has been proposed for this challenge…

> spoofing an agent seems in dubious territory.

Just to clarify, Perplexity is not spoofing a user agent, they're legitimately using a headless Chrome to fetch the page.

The author just misunderstood their docs [0]: when they say that "you can identify our web crawler by its user agent", they're talking about the crawler, not the browser they use for ad hoc requests. As you note, crawling is different.

[0] https://docs.perplexity.ai/docs/perplexitybot

Re: Perplexity AI is lying about their user agent

#297

Post by cal85 »

lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

> can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training.

I’m not saying you’re wrong, but why? And what do you mean by “your data” here?

Re: Perplexity AI is lying about their user agent

#298

Post by alexey-salmin »

mattigames wrote: post #270
JimDabell wrote: post #186

Earlier quoted context omitted.

Just to go a little bit more into detail on this, because the article and most of the conversation here is based on a big misunderstanding: robots.txt governs crawlers . Fetching a single user-specified URL is not crawling. Crawling is when you automatically follow links to continue fetching subsequent pages. Perplexity’s documentation that the article links to describes how their crawler works. That is not the piece…

Its fairly logical to assume that robots.txt governs robots (empahsis in "bots") not just crawlers, if they are only intended to block crawlers why aren't they called crawlers.txt instead and remove all ambiguity?

There's more than one way to define what a bot is.

You can make a request by typing the url in chrome, or by asking an AI tool to do so. Both start from user intent, both heavily rely on complicated software to work.

It's fairly logical to assume that bots don't have an intent and users do. It's not the only available interpretation though.

Re: Perplexity AI is lying about their user agent

#299

Post by marcus0x62 »

nradov wrote: post #278
marcus0x62 wrote: post #269

Earlier quoted context omitted.

Because if I run a server - at my own expense - I get to use information provided by the client to determine what, if any, response to provide? This isn’t a very difficult concept to grasp.

I'm having difficulty grasping the concept. Only a fool would trust any HTTP headers such as User-Agent sent by a random unauthenticated client. Your expenses are your problem.

… and I have absolutely no obligation to provide any particular response to any particular client.

Parsing, rendering, and trusting that the payload is consistent from request to request is your problem. You can connect to my server, or not. I really don’t care. What you cannot do is dictate how my server responds to your request.

Post reply on HN