61–70 of 555 posts
Earlier quoted context omitted.
Or detect the LLM and serve up an LLM rewritten version of the page. That way you feed it poisonous garbage.
The issue is detecting them when they use random user agents and ip ranges.
From what I've seen, most AI scrapers operate on known cloud IP ranges, usually amazon (Perplexity included), so just check for those.
> Next up is some kind of GDPR request perhaps? GDPR doesn't preclude anyone from scraping you. In fact, scraping is not illegal in any context (LinkedIn keeps losing lawsuits). Using copyrighted data in training LLMs is a huge grey area, but probably not outright illegal and will take like a decade (if not more) before we'll have legislative clarity.
If enough people demand it the effort gets too high and costly
Quibble with the headline-- I don't see a lie by Perplexity, they just aren't complying with a voluntary web standard.[1] [1] https://en.m.wikipedia.org/wiki/Robots.txt
The lie is in their documentation - they claim to use the PerplexityBot string in their user-agent: https://docs.perplexity.ai/docs/perplexitybot .
I think it is OK to use a different user agent for page retrievals made on demand that a user specifically requested (not to include in the index, just to answer a question).
But... I think that user agent should be documented and should not just be a browser default.
OpenAI do this for their crawlers: they have GPTBot for their crawler and ChatGPT-User for the requests made by their ChatGPT browser mode.
OpenAI scraped aggressively for years. Why should others put themselves behind an artificial moat? If you want to block access to a site, stop relying on arbitrary opt-in voluntary things like user agent or robots.txt. Make your site authenticated only, that’s literally the only answer here.
> OpenAI scraped aggressively for years. Why should others put themselves behind an artificial moat? Not saying I agree/disagree with the whole "LLMs trained on scraped data is unethical", but this way of thinking seems dangerous. If companies like Theranos can prop up their value by lying, does that make it ok for Theranos competitors to also lie, as another example?
What incentive does anybody have to be honest about their user agent?
Earlier quoted context omitted.
If using copyrighted material to train an LLM is theft, so is reading a book.
How is a human reading a book in any way related or comparable to a machine ingesting millions of books per day with the goal of stealing their content and replacing them?
humans learning is the old-school digital copying. computers simply do it much faster, but it's the same basic phenomenon
consider one teacher and one student. first there is one idea in one head but then the idea is in two heads.
now add book technology1 the teacher writes the book once, a thousand students read it. the idea has gone from being in one head (book author) onto most of the book readers!
Earlier quoted context omitted.
If using copyrighted material to train an LLM is theft, so is reading a book.
How is a human reading a book in any way related or comparable to a machine ingesting millions of books per day with the goal of stealing their content and replacing them?
(If we are to argue the latter point then it would also raise interesting implications; are we denying freedom of expression to a LLM when we fine-tune it or stop its generation?)
If you've ever tried to do any web scraping, you'll know why they lie about the User-Agent, and you'd do it too if you wanted your program to work properly. Discriminating based on User-Agent string is the unethical part.
> Why did you not respect robots.txt?
> I apologize for the mistake. I should have respected the robots.txt file for [my website], which likely disallows web scraping and crawling. I will make sure to follow the robots.txt guidelines in the future to avoid accessing restricted content.
Yeah, sure. What a joke.