Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity AI is lying about their user agent

rknight.me

Post reply on HN
Print view View original

41–50 of 555 posts

Re: Perplexity AI is lying about their user agent

#41

Post by brookst »

Mathnerd314 wrote: post #16
skilled wrote: post #5

Read this article if you want to know Perplexity’s idea of taking other people’s content and thinking they can get away with it, https://stackdiary.com/perplexity-has-a-plagiarism-problem/ The CEO said that they have some “rough edges” to figure out, but their entire product is built on stealing people’s content. And apparently[0] they want to start paying big publishers to make all that noise go away. [0]: https://w…

It's been debated at length, but to make it short: piracy is not theft, and everyone in the LLM space has been taking other people’s content and so far getting away with it (pending lawsuits notwithstanding).

If using copyrighted material to train an LLM is theft, so is reading a book.

Re: Perplexity AI is lying about their user agent

#42

Post by supriyo-biswas »

JohnMakin wrote: post #9
wrs wrote: post #3

I don’t think we should lump together “AI company scraping a website to train their base model” and “AI tool retrieving a web page because I asked it to”. At least, those should be two different user agents so you have the option to block one and not the other.

Is it actually retrieving the page on the fly though? How do you know this? Even if it were - it’s not supposed to be able to.

You can just point it at a webserver and ask it a question like "Summarize the content at [URL]" with a sufficiently unique URL that no one would hit, maybe with an UUID. This is also explored on the very article itself.

In my testing they're using crawlers on AWS and they do not parse Javascript or CSS, so it is sufficient to serve some kind of interstitial challenge page like the one on Cloudflare, or you can build your own.

Re: Perplexity AI is lying about their user agent

#43

Post by bakugo »

brookst wrote: post #41
Mathnerd314 wrote: post #16

Earlier quoted context omitted.

It's been debated at length, but to make it short: piracy is not theft, and everyone in the LLM space has been taking other people’s content and so far getting away with it (pending lawsuits notwithstanding).

If using copyrighted material to train an LLM is theft, so is reading a book.

How is a human reading a book in any way related or comparable to a machine ingesting millions of books per day with the goal of stealing their content and replacing them?

Re: Perplexity AI is lying about their user agent

#45

Post by janalsncm »

teeray wrote: post #12
mirekrusin wrote: post #6

The only way out seems to be using obscene captcha.

Or detect the LLM and serve up an LLM rewritten version of the page. That way you feed it poisonous garbage.

The issue is detecting them when they use random user agents and ip ranges.

Re: Perplexity AI is lying about their user agent

#46

Post by zxxh »

skilled wrote: post #5

Read this article if you want to know Perplexity’s idea of taking other people’s content and thinking they can get away with it, https://stackdiary.com/perplexity-has-a-plagiarism-problem/ The CEO said that they have some “rough edges” to figure out, but their entire product is built on stealing people’s content. And apparently[0] they want to start paying big publishers to make all that noise go away. [0]: https://w…

[dead]

Re: Perplexity AI is lying about their user agent

#48

Post by tommy_axle »

Dwedit wrote: post #7

How about a trap URL in the Robots.txt file that triggers a 24 hour IP ban if you access it. If you don't want anyone innocent caught in the crossfire, you could make the triggering URL customized to their IP address.

IP banning might be limited if they're already using a proxy network, which is par nowadays for avoiding detection.

Re: Perplexity AI is lying about their user agent

#49

Post by bakugo »

jstanley wrote: post #17

If you've ever tried to do any web scraping, you'll know why they lie about the User-Agent, and you'd do it too if you wanted your program to work properly. Discriminating based on User-Agent string is the unethical part.

There is nothing unethical about not wanting AI companies to steal your content and sell it for a profit.
Post reply on HN