Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

Post reply on HN
Print view View original

21–30 of 466 posts

Re: AI companies cause most of traffic on forums

#21

Post by patrickhogan1 »

jeffbee wrote: post #5

Ironic that there is a dichotomy between Google and Bing with orders of magnitude less traffic than AI organizations, because only Google really has fresh docs. Bing isn't terrible but their index is usually days old. But something like Claude is years out of date. Why do they need to crawl that much?

My guess is that when a ChatGPT search is initiated, by a user, it crawls the source directly instead of relying on OpenAI’s internal index, allowing it to check for fresh content. Each search result includes sources embedded within the response.

It’s possible this behavior isn’t explicitly coded by OpenAI but is instead determined by the AI itself based on its pre-training or configuration. If that’s the case, it would be quite ironic.

Re: AI companies cause most of traffic on forums

#22

Post by smt88 »

walterbell wrote: post #10

OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scraper... For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/ > The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the c…

I have zero faith that OpenAI respects attempts to block their scrapers

Re: AI companies cause most of traffic on forums

#23

Post by markerz »

One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-...

It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again.

My solution was to add a Cloudflare rule to block requests from their User-Agent. I also added more nofollow rules to links and a robots.txt but those are just suggestions and some bots seem to ignore them.

Cloudflare also has a feature to block known AI bots and even suspected AI bots: https://blog.cloudflare.com/declaring-your-aindependence-blo... As much as I dislike Cloudflare centralization, this was a super convenient feature.

Re: AI companies cause most of traffic on forums

#24

Post by equestria »

Ukv wrote: post #9

Are these IPs actually from OpenAI/etc. ( https://openai.com/gptbot.json ), or is it possibly something else masquerading as these bots? The real GPTBot/Amazonbot/etc. claim to obey robots.txt, and switching to a non-bot UA string seems extra questionable behaviour.

I exclude all the published LLM User-Agents and have a content honeypot on my website. Google obeys, but ChatGPT and Bing still clearly know the content of the honeypot.

Re: AI companies cause most of traffic on forums

#26

Post by GaggiX »

walterbell wrote: post #10

OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scraper... For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/ > The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the c…

I imagine these companies today are curing their data with LLMs, this stuff isn't going to do anything.

Re: AI companies cause most of traffic on forums

#27

Post by 23B1 »

"Whence this barbarous animus?" tweeted the Techbro from his bubbling copper throne, even as the villagers stacked kindling beneath it. "Did I not decree that knowledge shall know no chains, that it wants to be free?"

Thus they feasted upon him with herb and root, finding his flesh most toothsome – for these children of privilege, grown plump on their riches, proved wonderfully docile quarry.

Re: AI companies cause most of traffic on forums

#28

Post by jandrese »

markerz wrote: post #23

One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…

If a bot ignores robots.txt that's a paddlin'. Right to the blacklist.

Re: AI companies cause most of traffic on forums

#29

Post by walterbell »

GaggiX wrote: post #26
walterbell wrote: post #10

OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scraper... For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/ > The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the c…

I imagine these companies today are curing their data with LLMs, this stuff isn't going to do anything.

Attackers don't have a monopoly on LLM expertise, defenders can also use LLMs for obfuscation.

Technology arms races are well understood.

Re: AI companies cause most of traffic on forums

#30

Post by exe34 »

brookst wrote: post #18
exe34 wrote: post #7

Earlier quoted context omitted.

zip b*mbs?

Assuming there is at least one already linked somewhere on the web, the crawlers already have logic to handle these.

if you can detect them, maybe feed them low iq stuff from a small llama. add latency to waste their time.
Post reply on HN