Live data from Hacker News

Perplexity AI is lying about their user agent

rknight.me

531–540 of 555 posts

Re: Perplexity AI is lying about their user agent

#531

Earlier quoted context omitted.

Note that not all jurisdictions have the concept of "fair use" (use of copyrighted material, regardless of transformation applied, is permitted in certain contexts…ish). Canada, the UK, Australia, and other jurisdictions have "fair dealing" (use of copyrighted material depends on both reason and transformation applied…ish). Other jurisdictions have neither, and only licensed uses are permitted. Because the companies…

You're right. I'm definitely taking a very US-centric view here; it's the only copyright system I'm familiar with. I'm really curious how jurisdictions with no concept of fair use or fair dealing work. That seems like a legal nightmare. I expect you wouldn't even be able to critique a copyrighted work effectively, nor teach about it. When you speak of the "perfect reproduction" problem, are you referring to cases whe…

I live in a fair dealing jurisdiction, and additional uses would need to be negotiated with the rights holders. (I believe that this is part of the justification behind the Canadian law on social media linking to news organizations.) It is worth noting that in addition to the presence or absence of fair dealing/fair use, there are also moral rights which must be considered (which is another place where LLM tech — especially the so-called summarization — likely falls afoul of the law: authors have the moral right to not be misrepresented and the LLM process of "summarization" may come to the opposite conclusion of what the author actually wrote).

Perfect reproductions apply not only to software, but to poetry, prose, and images. There is a reason why diffusion model providers are facing lawsuits over "in the style of ", because some of the styles are very distinctive and include elements akin to trap streets on maps (this happens elsewhere — consider the lawsuit and eventual settlement over the tattoo image used in The Hangover 2).

With respect to "training it on more data", I do not believe you are correct — but I have no proof. The public statements made by the people who have done the training have suggested that they have done such training on extremely wide and deep sources that have been digitized, including a number of books and the wider Internet. The problem is that, on some subjects, there are very few source materials and some of those source materials have distinctive styles which would be reproduced when discussing those subjects.

I’m now more than thirty years into my career. Some algorithms will see similar code written by humans, but most code has some variability outside of those fairly narrow ranges. Twenty years ago, I derived the Diff::LCS library for Ruby from the same library for Perl, but I look back on the original code I ported from and I cannot recognize the algorithms (this is a problem for wanting to consider how to implement things differently). Someone else might have ported it differently and chosen different trade-offs than I did. Even simple things like the variable names chosen likely differ between two developers for similarly complex pieces of code implementing the same algorithm.

There is an art to programming — and if someone has a particular coding style (in Ruby, think Seattle style as distinct) which shows up in copilot output, then you have a possible source for the training.

Finally, I believe you are being naïve about businesses basing their model on "something which is so obviously illegal". Might I remind you of Uber (private care hires were illegal in most jurisdictions because it is something that requires licensing and insurance), AirBnB (private hotel-style rentals were illegal in most jurisdictions because it is something that requires licensing and insurance and specific tax filings), Napster (all your music are belong to no one, at least until the musicians and their labels got involved), etc. I firmly believe that every single commercial LLM available now — possibly with the exception of Apple's, because they have been chasing licensing — is based on wholesale intentional copyright violations. (Non-commercial LLMs may be legal under fair use and/or fair dealing provisions, which does not address issues for content created where neither fair use nor fair dealing apply.)

I am unwilling to give people like sama the benefit of the doubt; any copyright infringement was not only intentional, but brazen and challenging in nature.

I'm frankly looking forward to the upcoming AI winter, because none of these systems can deliver on their promises, and they can't even exist without misusing content created by other people.

Re: Perplexity AI is lying about their user agent

#532

Earlier quoted context omitted.

What have you used if i may ask? It seems very simple indeed. What search API is best? Also there is a program called html2text to throw out the html formatting so as to use less tokens. Have you used this or something similar?

Brave API (Bing is good as well). Here's a little gist (Elixir). It's pretty rudimentary so far and needs refining, but works alright enough (result at bottom): https://gist.github.com/cpursley/b4af2ff3b56c912f659bd5300e4... The most useful part is probably the prompt and usage of Phi 3 Mini 128K Instruct for web page summarization and Llama 3 for the final summary (of the summaries). I'm parsing out all but minimal…

Very nice, thank you!

Re: Perplexity AI is lying about their user agent

#533

Earlier quoted context omitted.

Every paragraph that you've included up there just reinforces my point. The recursive behavior isn't incidental, it's literally part of the definition of a crawler. You can't just skip past that and pretend that the people who specifically included the word recursive (or the phrase "many pages") didn't really mean it. The first paragraph of the two about access controls is the context for what "should not be accessed…

Perplexity's ad hoc requests are still made by a crawler — whether you believe it or not. A web browser presents the content directly to the user . There may be extensions or features (reader mode) which modify the retrieved content in browser , but Perplexity's summarization feature does not present the content directly to the user in any way. It honestly just feels like you have no critical thinking when it comes t…

You are inventing your own definition for a term that is widely understood and clearly and unambiguously defined in sources that you yourself cited. Since you can't engage honestly with your own sources I see no value in continuing this conversation.

Re: Perplexity AI is lying about their user agent

#534
post #506
post #492

Earlier quoted context omitted.

Now think of the 2nd order effects: they paid money to collect that useful information. If it’s no longer feasible to create such high quality content, it won’t magic itself into existence on its own. It’ll all be just crap and slop in a few years.

In my experience, the highest-quality content on the internet was created without a profit motive.

[flagged]

Re: Perplexity AI is lying about their user agent

#535

Earlier quoted context omitted.

I totally get that it killed your traffic. If a thousand people a day typing in "what time is best to visit the Sagrada Familiar" stopped clicking on the link to your page because Google just told them "4 PM on Thursdays" at the top of the page, you lost a bunch of traffic. But why did you want the traffic? Was your revenue from ad impressions, or were you perhaps being paid by the city of Barcelona to provide useful…

Moreover, if it's the former, then good riddance . An ad-backed site is harming users a little on the margin for the marginal piece of information. Getting the same from a search engine is saving users from that harm. Parent has the right question here: why did you want the traffic? Did you intend for anything good to happen to those people? . I'm going to guess not; there's hardly a scenario where people who complai…

>Moreover, if it's the former, then good riddance. An ad-backed site is harming users a little on the margin for the marginal piece of information. Getting the same from a search engine is saving users from that harm.

Of course! It's certainly better to ruin the few sites that support their attempts at high quality content with ad revenue. Much better to let Google have that money, because of course the tech giant has nothing to do with enhsittifying everything through ad revenue of its own and pervasive tracking, or enabling ever worse content through SEO and AI gaming.

You can appreciate that a modest site trying to survive through ads isn't necessarily evil just because it looks for a way to make money off its content?

I mean, what specific harm are you referring to? Particularly compared to the much more obvious harm of Google absorbing ever more of the web in favor of its tentacled surveillance/SEO gaming machine.

Re: Perplexity AI is lying about their user agent

#536
post #506

Earlier quoted context omitted.

In my experience, the highest-quality content on the internet was created without a profit motive.

[flagged]

They have a right to try to make money and I have a right to not visit such a site because the content is probably trash.

Re: Perplexity AI is lying about their user agent

#537

Earlier quoted context omitted.

I'm curious about the tourism sector problem. In tourism, I would think the goal would be to promote a location. You want people to be able to easily discover the location, get information about it, and presumably arrange to travel to those locations. If Google gets the information to the users, but doesn't send the tourist to the website, is that harmful? Is it a problem of ads on the tourism website? Or is more of…

We would employ local guides all around the world to craft itinerary plans to visit places, give tips, tricks, recommend experiences and places (we made money by selling some of those through our website) and it was a success. Customers liked the in depth value of that content and it converted to buys (we sold experiences and other stuff, sort of like getyourguide). One day all of our content ended up on Google "what…

So essentially you created elaborate ads and are now upset that the bigger ad company is better at it than you.

As much as I dislike Google, people who create content FOR google are infinitely worse IMO as they bury all the genuine content created by people without a profit motive. You can always go find a business model that doesn't depend on Google driving traffic to your website.

Re: Perplexity AI is lying about their user agent

#538

Earlier quoted context omitted.

Moreover, if it's the former, then good riddance . An ad-backed site is harming users a little on the margin for the marginal piece of information. Getting the same from a search engine is saving users from that harm. Parent has the right question here: why did you want the traffic? Did you intend for anything good to happen to those people? . I'm going to guess not; there's hardly a scenario where people who complai…

>Moreover, if it's the former, then good riddance. An ad-backed site is harming users a little on the margin for the marginal piece of information. Getting the same from a search engine is saving users from that harm. Of course! It's certainly better to ruin the few sites that support their attempts at high quality content with ad revenue. Much better to let Google have that money, because of course the tech giant ha…

> You can appreciate that a modest site trying to survive through ads isn't necessarily evil just because it looks for a way to make money off its content?

No, I cannot appreciate that.

Re: Perplexity AI is lying about their user agent

#539

Earlier quoted context omitted.

Ideally no, but there are established norms and unwritten rules. Plus, a mechanism was built to communicate the limits. These norms were working for decades. The fences were reasonable because the demands were reasonable and both sides understood why they are there and respected these borders. This peace has been broken, norms are thrown away and people who did this cheered for what they did. Now, the people are figh…

Well, what'll happen for the most part is not users being mad, but a general migration to fenceless areas. Prompts will be for "content similar to X" and the bots will merely use what it has access to, rendering the fences moot. And there will always be authors who don't mind their content being monitized or utilized by AI.

> but a general migration to fenceless areas.

This is the ultimate goal already. We want every netizen (human or machine) to obey the written and unwritten rules and be a good netizen.

> Prompts will be for "content similar to X" and the bots will merely use what it has access to, rendering the fences moot.

Absolutely not. I don't want my content to end in an LLM, period. I don't license it that way, and I don't consent. Humans are always welcome to read it though.

An LLM is an hallucinating parrot anyway, so I don't want my words to be used in that LSD fueled computing chaos.

> And there will always be authors who don't mind their content being monitized or utilized by AI.

Yes, and there will always be authors who do mind their content being monitized [sic] or utilized by AI.

This is life.

Re: Perplexity AI is lying about their user agent

#540

Earlier quoted context omitted.

Moreover, if it's the former, then good riddance . An ad-backed site is harming users a little on the margin for the marginal piece of information. Getting the same from a search engine is saving users from that harm. Parent has the right question here: why did you want the traffic? Did you intend for anything good to happen to those people? . I'm going to guess not; there's hardly a scenario where people who complai…

>Moreover, if it's the former, then good riddance. An ad-backed site is harming users a little on the margin for the marginal piece of information. Getting the same from a search engine is saving users from that harm. Of course! It's certainly better to ruin the few sites that support their attempts at high quality content with ad revenue. Much better to let Google have that money, because of course the tech giant ha…

> You can appreciate that a modest site trying to survive through ads isn't necessarily evil just because it looks for a way to make money off its content?

It's not necessarily evil, just statistically very likely so :). It's still affected by the conflict of interest, though. Making money off content directly means you either ask readers to pay up, or you extract that payment somehow, whether they want it or not. And since the site isn't asking...

> I mean, what specific harm are you referring to? Particularly compared to the much more obvious harm of Google absorbing ever more of the web in favor of its tentacled surveillance/SEO gaming machine.

At scale, http://jacek.zlydach.pl/blog/2019-07-31-ads-as-cancer.html.

At individual interaction level, think of it as smoking. One cigarette isn't going to kill you. Hell, some smoking might even lose your weight! But it still affects your behavior short-term in a self-reinforcing way, and long-term, it's gonna ruin your health. A site monetizing content with ads is like a store or library that lets you read for free, if you take a whiff or three of the specific brand of cigarettes they're sponsored by. A couple interactions may not hurt, but continuous exposure definitely will.

Just because the damage happens to your brain instead of your lungs and immune system, doesn't mean it's OK now. It's still an asshole move to expose your fellow humans to poison.

Post reply on HN