Sites scramble to block ChatGPT web crawler after instructions emerge
1–10 of 38 posts
Re: Sites scramble to block ChatGPT web crawler after instructions emerge
#2> blocking GPTBot will not guarantee that a site's data does not end up training all AI models of the future. Aside from issues of scrapers ignoring robots.txt files, there are other large data sets of scraped websites (such as The Pile) that are not affiliated with OpenAI.
This is why I'm not reassured. robots.txt isn't sufficient to stop all webcrawlers, so there every reason to think it isn't sufficient to stop AI scrapers.
I'm still wanting to find a good solution to this problem so that I can open my sites up to the public again.
Re: Sites scramble to block ChatGPT web crawler after instructions emerge
#3I wonder if its worth poisoning the replies for scrapers that don't obey robots.txt. Send back nonsense, lies, and noise. This would be an adversarial approach like https://adnauseam.io/ uses for ad tracking.
Re: Sites scramble to block ChatGPT web crawler after instructions emerge
#4The article does not say whether it obeys `User-agent: *`. My guess is that, if it doesn't respect that, it doesn't truly respect `User-agent: GPTBot` either.
Re: Sites scramble to block ChatGPT web crawler after instructions emerge
#5blocked it on every single site I manage
there is zero benefit to me in allowing OpenAI to absorb my content
it is a parasite, plain and simple (as is GitHub Copilot)
and I'll be hooking in the procedurally generated garbage pages for it soon!
Re: Sites scramble to block ChatGPT web crawler after instructions emerge
#6> blocking GPTBot will not guarantee that a site's data does not end up training all AI models of the future. Aside from issues of scrapers ignoring robots.txt files, there are other large data sets of scraped websites (such as The Pile) that are not affiliated with OpenAI. This is why I'm not reassured. robots.txt isn't sufficient to stop all webcrawlers, so there every reason to think it isn't sufficient to stop AI…
There's never going to be a perfect solution, it's an arms race. I really doubt (hope?) that large entities are going to straight up emulate end-user browsers though.
I would think filtering based on user agent will be the sweet spot for effort and performance. You could do some awful JavaScript monstrosity to detect the tiny fraction of bots who are sneaky, but if they're determined to be sneaky they will succeed at scraping.
Re: Sites scramble to block ChatGPT web crawler after instructions emerge
#7I've been reading lots of datasheets and application notes in the embedded space recently. Most of these are only accessible after creating a (free) login. In one sense, it's a reasonably simple way to prevent scraping like this (at least until the AI-based scrapers can generate their own logins). On the other hand, a lot of that kind of material would be _really_ useful to be able to ask an LLM about.
Re: Sites scramble to block ChatGPT web crawler after instructions emerge
#8blocked it on every single site I manage there is zero benefit to me in allowing OpenAI to absorb my content it is a parasite, plain and simple (as is GitHub Copilot) and I'll be hooking in the procedurally generated garbage pages for it soon!
What if it becomes the next Google? Are you really sure you want to be removed from their index?
Re: Sites scramble to block ChatGPT web crawler after instructions emerge
#9blocked it on every single site I manage there is zero benefit to me in allowing OpenAI to absorb my content it is a parasite, plain and simple (as is GitHub Copilot) and I'll be hooking in the procedurally generated garbage pages for it soon!
What if it becomes the next Google? Are you really sure you want to be removed from their index?
As it is, as GP says there would be 'zero benefit' to being included in that new Google, to GP as content author.
Re: Sites scramble to block ChatGPT web crawler after instructions emerge
#10> blocking GPTBot will not guarantee that a site's data does not end up training all AI models of the future. Aside from issues of scrapers ignoring robots.txt files, there are other large data sets of scraped websites (such as The Pile) that are not affiliated with OpenAI. This is why I'm not reassured. robots.txt isn't sufficient to stop all webcrawlers, so there every reason to think it isn't sufficient to stop AI…
I think bots are part of the public.