Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity AI is lying about their user agent

rknight.me

Post reply on HN
Print view View original

71–80 of 555 posts

Re: Perplexity AI is lying about their user agent

#71

Post by JoosToopit »

bakugo wrote: post #43
brookst wrote: post #41

Earlier quoted context omitted.

If using copyrighted material to train an LLM is theft, so is reading a book.

How is a human reading a book in any way related or comparable to a machine ingesting millions of books per day with the goal of stealing their content and replacing them?

Directly.

What if while reading you make notes - are you strealing content? If yes - should then people be forbidden from taking notes? How does writing down a note onto a piece of paper differ from writing it into your memory?

Re: Perplexity AI is lying about their user agent

#72

Post by fullspectrumdev »

Dwedit wrote: post #7

How about a trap URL in the Robots.txt file that triggers a 24 hour IP ban if you access it. If you don't want anyone innocent caught in the crossfire, you could make the triggering URL customized to their IP address.

This actually might work for fucking over certain web vulnerability scanners that will hit robots.txt to perform path/content discovery - have some trap urls that serve up deflate bombs and then ban the IP.

Re: Perplexity AI is lying about their user agent

#74

Post by Eisenstein »

bastawhiz wrote: post #53

I have a silly website that just proxies GitHub and scrambles the text. It runs on CF Workers. https://guthib.mattbasta.workers.dev For the past month or two, it's been hitting the free request limit as some AI company has scraped it to hell. I'm not inclined to stop them. Go ahead, poison your index with literal garbage. It's the cost of not actually checking the data you're indiscriminately scraping.

How does github feel about this? You are sending the traffic to them while changing the content.

Re: Perplexity AI is lying about their user agent

#76

Post by jsheard »

bakugo wrote: post #61
janalsncm wrote: post #45

Earlier quoted context omitted.

The issue is detecting them when they use random user agents and ip ranges.

> when they use random user agents and ip ranges From what I've seen, most AI scrapers operate on known cloud IP ranges, usually amazon (Perplexity included), so just check for those.

I assume that's why Reddit appears to be cracking down on VPNs lately, they probably don't actually care about VPNs but they're throttling scraper traffic coming from datacenter IP address ranges, which VPN providers are also using.

Re: Perplexity AI is lying about their user agent

#77

Post by bombela »

sjm-lbm wrote: post #31
unyttigfjelltol wrote: post #13

Quibble with the headline-- I don't see a lie by Perplexity, they just aren't complying with a voluntary web standard.[1] [1] https://en.m.wikipedia.org/wiki/Robots.txt

The lie is in their documentation - they claim to use the PerplexityBot string in their user-agent: https://docs.perplexity.ai/docs/perplexitybot .

It's not a lie. This is the agent string of the bot used for ingesting data for training the AI.

In the blog post, this is not what is happening. It is merely feeding the webpage as context to the AI during inference.

You are all confused here.

Re: Perplexity AI is lying about their user agent

#78

Post by jhbadger »

readyman wrote: post #21
skilled wrote: post #5

Read this article if you want to know Perplexity’s idea of taking other people’s content and thinking they can get away with it, https://stackdiary.com/perplexity-has-a-plagiarism-problem/ The CEO said that they have some “rough edges” to figure out, but their entire product is built on stealing people’s content. And apparently[0] they want to start paying big publishers to make all that noise go away. [0]: https://w…

> and thinking they can get away with it Can they not? I think that remains to be seen.

Exactly. It's like when Uber started and flaunted the medallion taxi system of many cities. People said "These Uber people are idiots! They are going to get shut down! Don't they know the laws for taxis?" While a small number of cities did ban Uber (and even that generally only temporarily), in the end Uber basically won. I think a lot of people confuse what they want to happen versus what will happen.

Re: Perplexity AI is lying about their user agent

#79

Post by snyp »

brookst wrote: post #41
Mathnerd314 wrote: post #16

Earlier quoted context omitted.

It's been debated at length, but to make it short: piracy is not theft, and everyone in the LLM space has been taking other people’s content and so far getting away with it (pending lawsuits notwithstanding).

If using copyrighted material to train an LLM is theft, so is reading a book.

But you pay money to buy a book and read it.

Re: Perplexity AI is lying about their user agent

#80

Post by sjm-lbm »

simonw wrote: post #63
sjm-lbm wrote: post #31

Earlier quoted context omitted.

The lie is in their documentation - they claim to use the PerplexityBot string in their user-agent: https://docs.perplexity.ai/docs/perplexitybot .

That is for the crawler, which is used to collect data for their search index. I think it is OK to use a different user agent for page retrievals made on demand that a user specifically requested (not to include in the index, just to answer a question). But... I think that user agent should be documented and should not just be a browser default. OpenAI do this for their crawlers: they have GPTBot for their crawler an…

Yeah, that seems reasonable to me as well. I'm honestly not sure if this is a "lie" in the most basic sense, or more information omission done in a way that feels intentionally dishonest.

At the very least, I do think that having an entire page in your docs about the user-agent strings you use without mentioning that, sometimes, you don't use those user agents at all is fairly misleading.

Post reply on HN