51–60 of 555 posts
Earlier quoted context omitted.
Or detect the LLM and serve up an LLM rewritten version of the page. That way you feed it poisonous garbage.
I really like this idea. Someone needs to implement this. I'm not sure what the ideal poison would be. Randomly constructed sentences that follow the basic rules of grammar?
If you've ever tried to do any web scraping, you'll know why they lie about the User-Agent, and you'd do it too if you wanted your program to work properly. Discriminating based on User-Agent string is the unethical part.
If I know the creator of the page doesn't want his page used by my program I wouldn't do it.
>Discriminating based on User-Agent string is the unethical part.
Not being exploited by an AI company is unethical? Robots.txt is made for telling bot identified by user agent what they are allowed to read.
https://guthib.mattbasta.workers.dev
For the past month or two, it's been hitting the free request limit as some AI company has scraped it to hell. I'm not inclined to stop them. Go ahead, poison your index with literal garbage. It's the cost of not actually checking the data you're indiscriminately scraping.
I don’t think we should lump together “AI company scraping a website to train their base model” and “AI tool retrieving a web page because I asked it to”. At least, those should be two different user agents so you have the option to block one and not the other.
Is it actually retrieving the page on the fly though? How do you know this? Even if it were - it’s not supposed to be able to.
They are able to do so.
> How do you know this?
The access logs.
> Even if it were - it’s not supposed to be able to.
There is a distinction from data used to train a model, which is the indexing bot with the custom user-agent string, and the user-query input given to the aforementioned AI model. When you ask an AI some question, you normally input text into a form, and the text goes back to the AI model where the magic happens. In this scenario, instead of inputting a wall text into a form, the text is coming from a url.
These forms of user input are equivilent, and yet distinctly different. Therefore it's intelectually dishonest for the OP to claim the AI is indexing them, when OP is asking the AI to fetch their website to augment or add context to the question being asked.
Read this article if you want to know Perplexity’s idea of taking other people’s content and thinking they can get away with it, https://stackdiary.com/perplexity-has-a-plagiarism-problem/ The CEO said that they have some “rough edges” to figure out, but their entire product is built on stealing people’s content. And apparently[0] they want to start paying big publishers to make all that noise go away. [0]: https://w…
It's been debated at length, but to make it short: piracy is not theft, and everyone in the LLM space has been taking other people’s content and so far getting away with it (pending lawsuits notwithstanding).
That’s a hell of a caveat!
Earlier quoted context omitted.
Can’t wait for OpenAI to settle with The New York Times. For a billion dollars no less.
Only reason OpenAI would do that would be to create a barrier for smaller entrants.
Only? No. Not even main.
The main reason would be to halt discovery and setting a precedent that would fuel not only further litigation but also, potentially, legislation.
That said, OpenAI should spin it as that master-of-the-universe take.
Quibble with the headline-- I don't see a lie by Perplexity, they just aren't complying with a voluntary web standard.[1] [1] https://en.m.wikipedia.org/wiki/Robots.txt
Earlier quoted context omitted.
It's been debated at length, but to make it short: piracy is not theft, and everyone in the LLM space has been taking other people’s content and so far getting away with it (pending lawsuits notwithstanding).
If using copyrighted material to train an LLM is theft, so is reading a book.
If you've ever tried to do any web scraping, you'll know why they lie about the User-Agent, and you'd do it too if you wanted your program to work properly. Discriminating based on User-Agent string is the unethical part.
News at 11.