Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity AI is lying about their user agent

rknight.me

Post reply on HN
Print view View original

101–110 of 555 posts

Re: Perplexity AI is lying about their user agent

#101

Post by simonw »

sjm-lbm wrote: post #80
simonw wrote: post #63

Earlier quoted context omitted.

That is for the crawler, which is used to collect data for their search index. I think it is OK to use a different user agent for page retrievals made on demand that a user specifically requested (not to include in the index, just to answer a question). But... I think that user agent should be documented and should not just be a browser default. OpenAI do this for their crawlers: they have GPTBot for their crawler an…

Yeah, that seems reasonable to me as well. I'm honestly not sure if this is a "lie" in the most basic sense, or more information omission done in a way that feels intentionally dishonest. At the very least, I do think that having an entire page in your docs about the user-agent strings you use without mentioning that, sometimes, you don't use those user agents at all is fairly misleading.

Yeah, I agree with that.

Re: Perplexity AI is lying about their user agent

#102

Post by rknightuk »

janalsncm wrote: post #33
JohnMakin wrote: post #9

Earlier quoted context omitted.

Is it actually retrieving the page on the fly though? How do you know this? Even if it were - it’s not supposed to be able to.

To steel man this, even though I think the article did a fine job already, maybe the author could’ve changed the content on the page so you would know if they were serving a cached response.

Author here. The page I asked it to summarize was posted after I implemented all blocking on the server (and robots.txt). So they should not have had any cached data.

Re: Perplexity AI is lying about their user agent

#103

Post by ai4ever »

IAmGraydon wrote: post #23
teeray wrote: post #12

Earlier quoted context omitted.

Or detect the LLM and serve up an LLM rewritten version of the page. That way you feed it poisonous garbage.

I really like this idea. Someone needs to implement this. I'm not sure what the ideal poison would be. Randomly constructed sentences that follow the basic rules of grammar?

make this open-source, I can bet you will see a lot of contributors.

then, make it easy for content producers to incorporate into their websites.

Re: Perplexity AI is lying about their user agent

#104

Post by lolinder »

Mathnerd314 wrote: post #16
skilled wrote: post #5

Read this article if you want to know Perplexity’s idea of taking other people’s content and thinking they can get away with it, https://stackdiary.com/perplexity-has-a-plagiarism-problem/ The CEO said that they have some “rough edges” to figure out, but their entire product is built on stealing people’s content. And apparently[0] they want to start paying big publishers to make all that noise go away. [0]: https://w…

It's been debated at length, but to make it short: piracy is not theft, and everyone in the LLM space has been taking other people’s content and so far getting away with it (pending lawsuits notwithstanding).

> so far getting away with it (pending lawsuits notwithstanding).

I know it feels like it's been longer, but it's not even been 2 years since ChatGPT was released. "So far" is in fact a very short amount of time in a world where important lawsuits like this can take 11 years to work their way through the courts [0].

[0] https://en.m.wikipedia.org/wiki/Oracle_v_Google

Re: Perplexity AI is lying about their user agent

#105

Post by jhbadger »

seanhunter wrote: post #93
jhbadger wrote: post #78

Earlier quoted context omitted.

Exactly. It's like when Uber started and flaunted the medallion taxi system of many cities. People said "These Uber people are idiots! They are going to get shut down! Don't they know the laws for taxis?" While a small number of cities did ban Uber (and even that generally only temporarily), in the end Uber basically won. I think a lot of people confuse what they want to happen versus what will happen.

In London, uber did not succeed. Uber drivers have to be licensed like minicab drivers.

Perhaps. But a reasonable license requiring you to pass a test isn't the same as a medallion in the traditional American taxi system. Medallions (often costing tens or even hundreds of thousands of dollars) were a way of artificially reducing the number of taxis (and thus raising the price).

Re: Perplexity AI is lying about their user agent

#106

Post by more_corn »

You should complain to their cloud host that they are knowingly stealing your content (because they’re hiding their user agent). Get them kicked off their provider for violating TOS. The CCPA also allows you to request that they delete your data. As a California company they have to comply or face serious fines.

Re: Perplexity AI is lying about their user agent

#107

Post by esha_manideep »

bastawhiz wrote: post #53

I have a silly website that just proxies GitHub and scrambles the text. It runs on CF Workers. https://guthib.mattbasta.workers.dev For the past month or two, it's been hitting the free request limit as some AI company has scraped it to hell. I'm not inclined to stop them. Go ahead, poison your index with literal garbage. It's the cost of not actually checking the data you're indiscriminately scraping.

They check after they scrape

Re: Perplexity AI is lying about their user agent

#108

Post by rknightuk »

maxrmk wrote: post #84

The author has misunderstood when the perplexity user agent applies. Web site owners shouldn’t dictate what browser users can access their site with - whether that’s chrome, firefox, or something totally different like perplexity. When retrieving a web page _for the user_ it’s appropriate to use a UA string that looks like a browser client. If perplexity is collecting training data in bulk without using their UA that…

It’s not retrieving a web page though is it? It’s retrieving the content then manipulating it. Perplexity isn’t a web browser.

Re: Perplexity AI is lying about their user agent

#110

Post by monocasa »

JumpCrisscross wrote: post #56
brookst wrote: post #38

Earlier quoted context omitted.

Only reason OpenAI would do that would be to create a barrier for smaller entrants.

> Only reason OpenAI would do that would be to create a barrier for smaller entrants Only? No. Not even main. The main reason would be to halt discovery and setting a precedent that would fuel not only further litigation but also, potentially, legislation. That said, OpenAI should spin it as that master-of-the-universe take.

A billion dollar settlement is more than enough to fuel further litigation.
Post reply on HN