Hi I am a blogger working on a small niche . I have written all these articles from the ground up with considerable effort. I don't want this to end up just as a free training data set for LLMs. Is there anything I can do to prevent that and still keep my site open free for visitors?
How can I prevent my site from being a free dataset for LLMs?
1–10 of 66 posts
Re: How can I prevent my site from being a free dataset for LLMs?
#2I think there is a noai meta tag
Re: How can I prevent my site from being a free dataset for LLMs?
#3For ChatGPT you can block it as per https://platform.openai.com/docs/plugins/bot
For others, take same measures as unwanted traffic or scrapping.
Re: How can I prevent my site from being a free dataset for LLMs?
#4For ChatGPT you can block it as per https://platform.openai.com/docs/plugins/bot For others, take same measures as unwanted traffic or scrapping.
Those instructions seem to be for plugins, not scraping training data.
In any case, OpenAI should inspect every website's terms of use before ingesting it in their training data. They shouldn't be exempted from this work. We shouldn't have to conform to their methods, there are laws and systems in place for that. Expensive, yes.
Re: How can I prevent my site from being a free dataset for LLMs?
#5The training on GPT was done on Common Crawl, Reddit, books, and Wikipedia.
For Common Crawl, the documentation says blocking it on robots.txt should work, as for Wikipedia, Reddit, and books, there's no option than to not participate AFAIK.
OpenWebText2 has no mention of robots.txt, so good luck with that.
Re: How can I prevent my site from being a free dataset for LLMs?
#6Require a login to read beyond the first paragraph
Re: How can I prevent my site from being a free dataset for LLMs?
#7[dead]
Re: How can I prevent my site from being a free dataset for LLMs?
#8Require a login to read beyond the first paragraph
This is the only way. Yes, a reputable company operating in a country that respects laws will respect your robots.txt or some sort of future no-ai tag. But everyone else won't.
Re: How can I prevent my site from being a free dataset for LLMs?
#9For ChatGPT you can block it as per https://platform.openai.com/docs/plugins/bot For others, take same measures as unwanted traffic or scrapping.
It should be take permission first instead of this obscenity
Re: How can I prevent my site from being a free dataset for LLMs?
#10I think there is a noai meta tag
That's like expecting a "no dog on the lawn" sign to work on stray dogs.