Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity AI is lying about their user agent

rknight.me

Post reply on HN
Print view View original

221–230 of 555 posts

Re: Perplexity AI is lying about their user agent

#221

Post by bastawhiz »

Eisenstein wrote: post #74
bastawhiz wrote: post #53

I have a silly website that just proxies GitHub and scrambles the text. It runs on CF Workers. https://guthib.mattbasta.workers.dev For the past month or two, it's been hitting the free request limit as some AI company has scraped it to hell. I'm not inclined to stop them. Go ahead, poison your index with literal garbage. It's the cost of not actually checking the data you're indiscriminately scraping.

How does github feel about this? You are sending the traffic to them while changing the content.

Frankly I don't care. They can block me if they want.

Re: Perplexity AI is lying about their user agent

#222

Post by insane_dreamer »

bmicraft wrote: post #70
surfingdino wrote: post #58

Earlier quoted context omitted.

Reading a book is not theft. Building a business on processing other people's copyrighted material to produce content is.

I think that's called a school

Schools pay for books, or use public domain materials

Re: Perplexity AI is lying about their user agent

#223

Post by sebzim4500 »

mrweasel wrote: post #167
jhbadger wrote: post #78

Earlier quoted context omitted.

Exactly. It's like when Uber started and flaunted the medallion taxi system of many cities. People said "These Uber people are idiots! They are going to get shut down! Don't they know the laws for taxis?" While a small number of cities did ban Uber (and even that generally only temporarily), in the end Uber basically won. I think a lot of people confuse what they want to happen versus what will happen.

Uber is banned in multiple countries and pulled out of many more because they where told to follow the law and that makes their business unprofitable.

So? They have a market cap of $150 billion. If at the start they had decided "oh well let's not bother since what we are doing is legally ambiguous" they would have a market cap of $0.

Re: Perplexity AI is lying about their user agent

#224

Post by bastawhiz »

esha_manideep wrote: post #107
bastawhiz wrote: post #53

I have a silly website that just proxies GitHub and scrambles the text. It runs on CF Workers. https://guthib.mattbasta.workers.dev For the past month or two, it's been hitting the free request limit as some AI company has scraped it to hell. I'm not inclined to stop them. Go ahead, poison your index with literal garbage. It's the cost of not actually checking the data you're indiscriminately scraping.

They check after they scrape

That's a lot of time and bandwidth to waste

Re: Perplexity AI is lying about their user agent

#225

Post by ben_w »

cyanydeez wrote: post #147
Mathnerd314 wrote: post #16

Earlier quoted context omitted.

It's been debated at length, but to make it short: piracy is not theft, and everyone in the LLM space has been taking other people’s content and so far getting away with it (pending lawsuits notwithstanding).

Right, it's ironic we spent 30 years fighting piracy and then suddenly corporations start doing it and now it's suddenly ok.

For me, the irony is the opposite side of the same coin, 30 years of "information wants to be free" and "copyright infringement isn't piracy" and "if you don't want to be indexed, use robots.txt"…

…and then suddenly OpenAI are evil villains, and at least some of the people denounced them for copyright infringement are, in the same post, adamant that the solution is to force the model weights to become public domain.

Re: Perplexity AI is lying about their user agent

#226

Post by JumpCrisscross »

ulrikrasmussen wrote: post #179
cdme wrote: post #137

Earlier quoted context omitted.

How a user views a page isn't the same as a startup scraping the internet wholesale for financial gain.

But it's not scraping, it's retrieving the page on request from the user.

> it's not scraping, it's retrieving the page on request from the user

Search engines already tried it. It’s not retrieving on request because the user didn’t request the page, they requested a bot find specific content on any page.

Re: Perplexity AI is lying about their user agent

#227

Post by mrweasel »

croes wrote: post #62
dvt wrote: post #35

> Next up is some kind of GDPR request perhaps? GDPR doesn't preclude anyone from scraping you. In fact, scraping is not illegal in any context (LinkedIn keeps losing lawsuits). Using copyrighted data in training LLMs is a huge grey area, but probably not outright illegal and will take like a decade (if not more) before we'll have legislative clarity.

But per GDPR you could enforced your data fo be deleted. If enough people demand it the effort gets too high and costly

LLMs don't really retain the full data anyway and it "should" be scrapped once the training is done. So yes, technically you might be able to demand that your data is to be removed from the training data, but that's going to be fairly hard to prove that it exists within the model.

Re: Perplexity AI is lying about their user agent

#228

Post by emrah »

natch wrote: post #126

It seems to me there could be some confusion here. When providing a service such as Perplexity AI's, there are two use cases to consider for accessing web sites. One is the scraping use case for training, where a crawler is being used and it is gathering data in bulk. Hopefully in a way that doesn't hammer one site at a time, but spreads the requests around gently. The other is the use case for fulfilling a user's sp…

This

Re: Perplexity AI is lying about their user agent

#229

Post by alt227 »

lolinder wrote: post #200
putlake wrote: post #175

Earlier quoted context omitted.

It's funny I posted the inverse of this. As a web publisher, I am fine with folks using my content to train their models because this training does not directly steal any traffic. It's the "train an AI by reading all the books in the world" analogy. But what Perplexity is doing when they crawl my content in response to a user question is that they are decreasing the probability that this user would come to by content…

> A tool that runs on-device (like Reader mode) is different because Perplexity is an aggregator service that will continue to solidify its position as a demand aggregator and I will never be able to get people directly on my content. If I visit your site from Google with my browser configured to go straight to Reader Mode whenever possible, is my visit more useful to you than a summary and a link to your site provid…

Traffic numbers, regardless if it using reader mode or not, are used as a basic valuation of a website or page. This is why Alexa rankings have historically been so important.

If Perplexity visit the site once and cache some info to give to multiple users, that is stealing traffic numbers for ad value, but also taking away the ability from the site owner to get realistic ideas of how many people are using the information on their site.

Additionally, this is AI we are talking about. Whos to say that the genrated summary of information is actually correct? The only way to confirm that, or to get the correct information in the first place, is to read the original site yourself.

Post reply on HN