31–38 of 38 posts
From the article: > For example, blocking content from future AI models could decrease a site's or a brand's cultural footprint if AI chatbots become a primary user interface in the future. I would rather leave the internet entirely if AI chatbots become a primary user interface.
It's more pragmatic to expect that any data that can be accessed one way or another will be scraped because interests aren't aligned between content authors and scrapers.
On the other hand, robots.txt was benefiting both search engines and content authors because it signaled data that wasn't useful to show in search results, therefore search engines had an incentive to follow its rules.
This gives the illusion of being in control, but if enough people block the bot, they'll just scrape differently (if they don't already) because too much money is at stake, more than whatever fine they may get if they do get caught and can't settle out of court, not to mention they may consider it will be someone else's problem by then. It's more pragmatic to expect that any data that can be accessed one way or anoth…
Or perhaps all crawlers regardless of respecting robots. Honestly I am not interested in improving some FAANGish algorithm with blogposts intended for my friends.
For anyone reading this, you can skip the robots.txt, as others have pointed out, who knows if they will actually listen to it. Instead, use a redirect or return a response code by doing a user agent check in your server config. I posted elsewhere in this thread on the way i did it with nginx
...You realize user agent is operator set, right? If they won't reapect robots.txt, they aren't interested in your consent.
Respecting the robots.txt has nothing to do with what the UA is set to. Yes, you can say this UA can do x in the robots.txt, but not respecting it, makes it moot.
The method i put in place does not use robots.txt, so theres no need to worry about them not respecting it anymore.
As someone else mentioned, like the world of spam and such, its an arms race. The solution may not be perfect, but its functional
From the article: > For example, blocking content from future AI models could decrease a site's or a brand's cultural footprint if AI chatbots become a primary user interface in the future. I would rather leave the internet entirely if AI chatbots become a primary user interface.
Why? That would imply that they are pretty good for most things - better than what we have now.
Sugary cereals and desserts have taken over much of snacking today, doesn't mean it's a good thing.
> blocking GPTBot will not guarantee that a site's data does not end up training all AI models of the future. Aside from issues of scrapers ignoring robots.txt files, there are other large data sets of scraped websites (such as The Pile) that are not affiliated with OpenAI. This is why I'm not reassured. robots.txt isn't sufficient to stop all webcrawlers, so there every reason to think it isn't sufficient to stop AI…
I think bots are part of the public.
> blocking GPTBot will not guarantee that a site's data does not end up training all AI models of the future. Aside from issues of scrapers ignoring robots.txt files, there are other large data sets of scraped websites (such as The Pile) that are not affiliated with OpenAI. This is why I'm not reassured. robots.txt isn't sufficient to stop all webcrawlers, so there every reason to think it isn't sufficient to stop AI…
I chose to use an nginx entry, because i also dont trust them to follow robots.txt. Throwing a 410 Gone should keep them from coming back too, theoretically, assuming they actually eject when receiving it, like it should. `if ($http_user_agent ~* ".*?(GPTBot|AI).*?") { return 410; }` Its not perfect, but it should filter them indefinitely, will probably have to add some more terms in there over time.
> blocking GPTBot will not guarantee that a site's data does not end up training all AI models of the future. Aside from issues of scrapers ignoring robots.txt files, there are other large data sets of scraped websites (such as The Pile) that are not affiliated with OpenAI. This is why I'm not reassured. robots.txt isn't sufficient to stop all webcrawlers, so there every reason to think it isn't sufficient to stop AI…
There's never going to be a perfect solution, it's an arms race. I really doubt (hope?) that large entities are going to straight up emulate end-user browsers though. I would think filtering based on user agent will be the sweet spot for effort and performance. You could do some awful JavaScript monstrosity to detect the tiny fraction of bots who are sneaky, but if they're determined to be sneaky they will succeed at…
> if they're determined to be sneaky they will succeed at scraping.
Yes, which is why I suspect I will never be able to open my websites up to the general public again. I live in hope anyway.