Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity AI is lying about their user agent

rknight.me

Post reply on HN
Print view View original

371–380 of 555 posts

Re: Perplexity AI is lying about their user agent

#371

Post by bhelkey »

cal85 wrote: post #297
lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

> can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. I’m not saying you’re wrong, but why? And what do you mean by “your data” here?

> And what do you mean by “your data” here?

The website that they created.

Re: Perplexity AI is lying about their user agent

#372

Post by Zambyte »

zenlikethat wrote: post #272
Zambyte wrote: post #259

Earlier quoted context omitted.

> The only way to confirm that, or to get the correct information in the first place, is to read the original site yourself. As someone who uses Perplexity, I often do do this. And I don't think I'm particularly in the minority with this. I think their UI encourages it.

Yeah that's one of the best things about them for me. And then I go to the website and often it's some janky UI with content buried super deep. Or it's like Reddit and I immediately get slammed with login walls and a million annoying pop ups. So I'm quite grateful to have an ability to cut through the noise and non-consistency of the wild west web. I agree the idea that we're somewhat killing traffic to the organic w…

> I wonder if it would be slightly less sad if we all had our own decentralized crawlers that simply functioned as extensions of ourselves.

This is something I'm (slowly) working on myself. I have a local language model server and 30 tb usable storage ready to go, just working on the software :)

Re: Perplexity AI is lying about their user agent

#373

Post by lolinder »

mrweasel wrote: post #187
wrs wrote: post #3

I don’t think we should lump together “AI company scraping a website to train their base model” and “AI tool retrieving a web page because I asked it to”. At least, those should be two different user agents so you have the option to block one and not the other.

Personally I don't even think that the issue. I'd prefer correct user-agent, that just common decency and shouldn't be an issue for most. What I do expect the AI companies to do is to check the license of the content they scrape and follow that. Let's say I run a blog, and I have a CC BY-NC 4.0 license. You can train your AI and that content, as long as it's non-commercial. Otherwise you'd need to contact me an negot…

> I'd prefer correct user-agent, that just common decency and shouldn't be an issue for most.

Tell that to the Chrome team. And the Safari team. And the Opera team. [0]

[0] https://webaim.org/blog/user-agent-string-history/

Re: Perplexity AI is lying about their user agent

#374

Post by tensor »

marcus0x62 wrote: post #59
jstanley wrote: post #17

If you've ever tried to do any web scraping, you'll know why they lie about the User-Agent, and you'd do it too if you wanted your program to work properly. Discriminating based on User-Agent string is the unethical part.

Please explain - in detail - why using information communicated by the client to change how my server operates is “unethical”. Keep in mind I pay money and expend time to provide free content for people to consume.

Here is a simple example. If you made your website only work in say, Microsoft Edge, and blocked everyone else telling them to download Edge. I'd think you're an asshole. Whether or not being an ass is unethical I'll leave to the philosophers.

Clearly there are many other scenarios, and many that are more muddy, but overall when we get in to the business of trying to force people to consume content in particular ways it's a bit icky in my opinion.

The extreme end result of this is no more open web, just force people to download your app to consume your content. This is happening too and it sucks.

Re: Perplexity AI is lying about their user agent

#375

Post by bayindirh »

skeledrew wrote: post #154
bayindirh wrote: post #20

Earlier quoted context omitted.

What if the scraper is not respecting robots.txt to begin with? Aren't they unethical enough to warrant a stronger method to prevent scraping?

Should there be a difference in treatment between a user going on a website and manually copying the content over to a bot to process vs giving the bot the URL so it does the fetching as well? I've done both (mainly to get summaries or translations) and I know which I generally prefer.

Ideally no, but there are established norms and unwritten rules. Plus, a mechanism was built to communicate the limits. These norms were working for decades.

The fences were reasonable because the demands were reasonable and both sides understood why they are there and respected these borders.

This peace has been broken, norms are thrown away and people who did this cheered for what they did. Now, the people are fighting back. People were silent because the system was working.

It was akin to mark some doors "authorized personnel only" but leaving them unlocked. People and programs respected these stickers. Now there are people and programs who don't, so people started to reinforce these doors.

It doesn't matter what you prefer. The apples are spoiled now. There's no turning back. The days of peace and harmony are over, thanks to "move fast break things. We're doing something amazing anyway, and we don't no permission!" people. If your use is benign but my filter is preventing that use, you should get mad at the parties who caused this fence to appear. It's not my fault to put a fence to protect myself.

To see the current state of affairs, see this list [0]. I'm very sensitive to ethical issues about training your model with my data without my consent, and selling it to earn monies.

I don't care about how you stretch fair-use. The moment you earn money from your model, it's not fair-use anymore [1].

[0]: https://notes.bayindirh.io/notes/Lists/Discussions+about+Art...

[1]: https://news.ycombinator.com/item?id=39188979

Re: Perplexity AI is lying about their user agent

#376

Post by TeMPOraL »

KomoD wrote: post #344
BriggyDwiggs42 wrote: post #322

Earlier quoted context omitted.

I’d rather share less information than more to any site I visit. Why does a user want to share that info?

What, users won't share anything? I said I wanted Perplexity to identify themselves in the user agent instead of using the generic "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.3" they're using right now for the "non-scraper bot". How does that impact users at all?

I don't, because if it will, then someone like the author of the article will do the obnoxious thing and ban it. We've been there before, 30 years ago. That's why all browsers' user agent strings start with "Mozilla".

Re: Perplexity AI is lying about their user agent

#377

Post by baxtr »

ADeerAppeared wrote: post #332
baxtr wrote: post #183

Earlier quoted context omitted.

What will happen if: Website owners decide to stop publishing because it’s not rewarded by a real human visit anymore? Then perplexity and the like won’t have new information to train their models on and no sites to answer the questions. I think there is a real content dilemma here at work. The incentives of Google and website owners were more or less aligned. This is not the case with perplexity.

> I think there is a real content dilemma here at work It's not really a dilemma. This is exactly what copyright serves to protect authors from. Perplexity copied the content, and in doing so directly competes with the original work, destroying it's market value and driving the original author out of business. Literally what copyright was invented to prevent. It's the exact same situation as journalists going after G…

>And then what? AI needs humans creating original content, especially for things like journalism and fact-finding. It'd be an eternal AI winter, all LLMs doomed to be stuck in 2025.

It's in every AI developer's best interest to halt the likes of Perplexity immediately before they irreparably damage the field of AI.

That’s exactly the problem and we all know that it will happen.

Re: Perplexity AI is lying about their user agent

#378

Post by skeledrew »

marcus0x62 wrote: post #320
nradov wrote: post #310

Earlier quoted context omitted.

Sure. So just return an HTTP 4XX response to requests you don't like. What's the problem?

Or, I return whatever content I want, within the bounds of the law, based on whatever parameters I decide. What's your problem with that? Again, connect to my server or don't. But don't tell me what type of response I'm obligated to provide you. If I think a given request is from an LLM training module, I don't have any legal obligation whatsoever to return my original content. Or a 400-series response. If I want to…

This argument of freedom seems applicable on both sides. A site owner/admin is free to return whatever response they wish based on the assumed origin of a request. An LLM user/service is free to send whatever info in the request that elicits a useful response.

Re: Perplexity AI is lying about their user agent

#379

Post by baxtr »

TeMPOraL wrote: post #369
baxtr wrote: post #323

Earlier quoted context omitted.

A visit is a human reader. At the very least they get exposed to your website name. Notice your product/service if you get lucky. Become a customer at a later visit. We are talking about cutting the first step off so that everything which may come afterwards is cut off as well.

In other words, content is bait, reward is a captured user whose attention - whose sanity, the finite amount of life - can be wasted or plain used against them. I'm more than happy to see all the websites with attention economy business models to shut down. Yes, that might be 90% of the Internet. That would be the 90% that is poisonous shit.

The attention economy will never die. Attention will only shift. From websites to aggregators like perplexity.

Re: Perplexity AI is lying about their user agent

#380

Post by lolinder »

__loam wrote: post #352
lolinder wrote: post #130

There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…

The problem that Perplexity has that ad blockers don't is that they're an independent site that is publishing content based on work they didn't produce. That runs afoul of both copyright laws and section 230 which let's sites like Google and Facebook operate. That's pretty different from an ad blocker running on your local machine. The ad blocker isn't publishing the page it edited for you.

> they're an independent site that is publishing content based on work they didn't produce.

What distinguishes these two situations?

* User asks proprietary web browser to fetch content and render it a specific way, which it does

* User asks proprietary web service to fetch content and render it a specific way, which it does

The technical distinction is that there's a network involved in the second scenario. What is the moral distinction?

Why is it that a proprietary web service manipulating content on behalf of a user is "publishing" content illegally, while a proprietary web browser doing the exact same kind of transformations is not? Assume that in both cases the proprietary software fetches the data upon request, does not cache it, and does not make the transformed content available to other users.

Post reply on HN