151–160 of 555 posts
All of these AI Wrapper companies are getting pushed out of the market by big tech sooner or later. Those blue oceans are actually red as fuck.
I don’t think we should lump together “AI company scraping a website to train their base model” and “AI tool retrieving a web page because I asked it to”. At least, those should be two different user agents so you have the option to block one and not the other.
If an AI agent is performing a search on behalf of a user, should its user agent be the same as that user’s?
Perplexity's utility as an answer engine is RAG (retrieval augmented generation). In response to your question, they search the web, crawl relevant URLs and summarize them. They do include citations in their response to the user, but in practice no one clicks through on the tiny (1), (2) links to go to the source. So if you are one of those sources, you lose out on traffic that you would otherwise get in the old model from say a Google or Bing. When Perplexity crawls your web page in this context, they are hiding their identity according to OP, and there seems to be no way for publishers to opt out of this.
It is possible that when they crawl the web for the second use case -- to collect data for training their model -- they use the right user agent and identify themselves. A publisher may be OK with allowing their data to be crawled for use in training a model, because that use case does not directly "steal" any traffic.
If you've ever tried to do any web scraping, you'll know why they lie about the User-Agent, and you'd do it too if you wanted your program to work properly. Discriminating based on User-Agent string is the unethical part.
What if the scraper is not respecting robots.txt to begin with? Aren't they unethical enough to warrant a stronger method to prevent scraping?
Read this article if you want to know Perplexity’s idea of taking other people’s content and thinking they can get away with it, https://stackdiary.com/perplexity-has-a-plagiarism-problem/ The CEO said that they have some “rough edges” to figure out, but their entire product is built on stealing people’s content. And apparently[0] they want to start paying big publishers to make all that noise go away. [0]: https://w…
It's been debated at length, but to make it short: piracy is not theft, and everyone in the LLM space has been taking other people’s content and so far getting away with it (pending lawsuits notwithstanding).
Earlier quoted context omitted.
It's been debated at length, but to make it short: piracy is not theft, and everyone in the LLM space has been taking other people’s content and so far getting away with it (pending lawsuits notwithstanding).
If using copyrighted material to train an LLM is theft, so is reading a book.
No, they would sue me for everything I got, and then some. That's the weird thing about these companies, they are never afraid to use IP law to go after others, but those same laws don't apply to them... because?
Just pay the stupid license and if that makes your business unsustainable then it's not much a business is it?
I am not sure I will ever stop being weirded out, annoyed at, confused by, something... people asking these sorts of questions of an LLM. What, you want an apology out of the LLM?
Earlier quoted context omitted.
In London, uber did not succeed. Uber drivers have to be licensed like minicab drivers.
Perhaps. But a reasonable license requiring you to pass a test isn't the same as a medallion in the traditional American taxi system. Medallions (often costing tens or even hundreds of thousands of dollars) were a way of artificially reducing the number of taxis (and thus raising the price).
There are two different questions at play here, and we need to be careful what we wish for. The first concern is the most legitimate one: can I stop an LLM from training itself on my data? This should be possible and Perplexity should absolutely make it easy to block them from training. The second concern, though, is can Perplexity do a live web query to my website and present data from my website in a format that th…