Nice of them to respect crawling after they've already trained their model. Presumably these headers don't affect any pages they've already crawled to train GPT(?)
Their papers say they were using Common Crawl for crawling. If you didn't want your pages in Common Crawl (eg. Twitter didn't) for use in many downstream analyses or uses beyond just OA, you could already have said so in your robots.txt.
GPTBot – OpenAI’s Web Crawler
261–270 of 327 posts
Re: GPTBot – OpenAI’s Web Crawler
#262Earlier quoted context omitted.
> And it's important that we reject this definition because it distorts the reality of the situation. Depends who's reality. A content creator's reality is that their content is indeed stolen and monetised by someone without permission. "Advocates of expanded IP law" do appear to be in the right, at least by law. Copying and distributing digital products is treated more or less as theft, particularly when done at sca…
> Depends who's reality On a trivial level this is correct as words mean what we collectively decide they mean. However I am making the point that a) the meaning has been changed and b) it has changed in a way that is deceptive and masks a useful fact about the world
Correct, and collectively we decided that reselling digital work without permission is indeed theft, just as we rightfully decided that digital goods for the most part are like physical goods.
> a) the meaning has been changed
It hasn't really, digital theft still has the same meaning as any form of theft. Some did try to change the meaning and non trivialise the act based on the fact that digital goods are not like phyisical goods. But that's a techincallity based on the nature of digital goods.
Similarly, AI folks wish to change the meaning of theft based on the false assumption that an AI system "learns just like a human". But that's a false assumption. The software does mimic human behaviour, but we all know that it is neither human nor intelligent (if it were intelligent you'd show it a set of multiplications, and from that point onwards it would figure it out on its own. same with writing stories). Yet some are trying to change the meaning of words to accommodate their view of the world in which software that can ingest people's IP at massive scale, mix it in, and output something that looks novel is somehow similar to human learning.
Therefore the matter is trivial. Software ingestsing digital content without permission, and outputting content made of even tiny bits of the original, is theft. Simple as that. However, that does not mean that AI should be banned. It's how the AI software is fed its data that must be brought in line.
Re: GPTBot – OpenAI’s Web Crawler
#263Earlier quoted context omitted.
Are you considering ad revenue?
That already puts the website in the sleazy category. "I mixed my helpful information with mind poison" isn't a strong position to argue fair play from.
Re: GPTBot – OpenAI’s Web Crawler
#264Earlier quoted context omitted.
> Photoshop [...] runs in on a remote computer Does it? Last time I checked, “cloud” in “Creative Cloud” meant “now you have to pay a monthly subscription”. And reverse engineering Photoshop to make a competitor might be a legal practice, if done properly – for example, see the ReactOS project. Aside from that, I think your point still stands though.
Most of Photoshop works offline, but some of the newer 'AI' features run on Adobe servers and need an online connection (and account) to work.
Re: GPTBot – OpenAI’s Web Crawler
#265Earlier quoted context omitted.
Just one example: As a software vendor, you probably want OpenAI to index your documentation, so questions about your software can be answered by ChatGPT. Not everybody who creates content is a "content creator" (when did this word get the specific meaning "people who earn money or reputation from creating content").
That’s a good point, I hadn’t thought of these cases. On a second thought, I guess a lot of marketing content would also love to be crawled by anything that crawls…
Re: GPTBot – OpenAI’s Web Crawler
#266Yet another bot that completely ignores the "429 Too Many Requests" response header and happily continues hammering your tiny little side project [1] to death. Luckily, I already block the IP address they're using as it has been used for (other?) malicious bots before. [1] In my case, it relies on third-party APIs that are heavily rate limited. Any bot ignoring rate limitation measures will effectively (D)DOS my serv…
Yet another reason why you should handle these scenarios on your own rather than hoping clients/users will.
Re: GPTBot – OpenAI’s Web Crawler
#267Meanwhile... > For robots.txt, we do follow the same restrictions applied to googlebot, otherwise Google benefits from its dominant position. https://community.brave.com/t/stop-website-being-shown-in-br...
Re: GPTBot – OpenAI’s Web Crawler
#268What’s the incentive for people to allow the crawler at all? Unlike search engines, chatgpt doesn’t cite references at all (last I tried) or even if it does it often makes up nonexistent references. And because it rephrases the content, there’s often no way to prove they got the material from a particular source, so harder to litigate plagiarism too. How would contributing to the weights of this LLM help content crea…
Chatgpt 4 provides pretty good citations on request.
Re: GPTBot – OpenAI’s Web Crawler
#269Earlier quoted context omitted.
It doesnt memorize anything. It just needs gazillion parameters that approach the size of the training set to finesse its conversational accent.
LLama2 has a 5TB training set.
People dont want to acknowledge that the LLM structure reflects rather closely what it is being trained on, but the incredibly large number of parameters suggests it is closer to a photographic fit than a true abstraction. larger models being more likely to memorize training data (Carlini et al., 2021, 2022)
The fact that the information gets mangled and somewhat compressed doesnt change this close relationship.
Re: GPTBot – OpenAI’s Web Crawler
#270Setting up reverse DNS would be good rather than a dedicated page showing IP subnets.