31–40 of 466 posts
Are these IPs actually from OpenAI/etc. ( https://openai.com/gptbot.json ), or is it possibly something else masquerading as these bots? The real GPTBot/Amazonbot/etc. claim to obey robots.txt, and switching to a non-bot UA string seems extra questionable behaviour.
I exclude all the published LLM User-Agents and have a content honeypot on my website. Google obeys, but ChatGPT and Bing still clearly know the content of the honeypot.
Are these IPs actually from OpenAI/etc. ( https://openai.com/gptbot.json ), or is it possibly something else masquerading as these bots? The real GPTBot/Amazonbot/etc. claim to obey robots.txt, and switching to a non-bot UA string seems extra questionable behaviour.
I exclude all the published LLM User-Agents and have a content honeypot on my website. Google obeys, but ChatGPT and Bing still clearly know the content of the honeypot.
‘Tis why I only use Signal and private git and otherwise avoid “the open web” except via the occasional throwaway
It’s a naive college student project that spiraled out of control.
Should be careful before they get blacked and can’t get data anymore. ;)
OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scraper... For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/ > The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the c…
I have zero faith that OpenAI respects attempts to block their scrapers
OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scraper... For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/ > The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the c…
The latter is clever but unlikely to do any harm. These companies spend a fortune on pre-training efforts and doubtlessly have filters to remove garbage text. There are enough SEO spam pages that just list nonsense words that they would have to.
2. As a sibling proposes, this is probably going to become an perpetual arms race (even if a very small one in volume) between tech-savvy content creators of many kinds and AI companies scrapers.
One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…
If a bot ignores robots.txt that's a paddlin'. Right to the blacklist.
Ironic that there is a dichotomy between Google and Bing with orders of magnitude less traffic than AI organizations, because only Google really has fresh docs. Bing isn't terrible but their index is usually days old. But something like Claude is years out of date. Why do they need to crawl that much?
Earlier quoted context omitted.
court ruling a few years ago said it's legal to scrape web pages, you don't need to be respectful of these for any purely legal reasons however this doesn't stop the website from doing what they can to stop scraping attempts, or using a service to do that for them
> court ruling Isn't this country dependent though?