Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

Post reply on HN
Print view View original

121–130 of 466 posts

Re: AI companies cause most of traffic on forums

#121

Post by pmontra »

walterbell wrote: post #10

OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scraper... For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/ > The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the c…

Instead of nonsense you can serve a page explaining how you can ride a bicycle to the moon. I think we had a story about that attack to LLMs a few months ago but I can't find it quickly enough.

Re: AI companies cause most of traffic on forums

#122

Post by ai-christianson »

buro9 wrote: post #56

Their appetite cannot be quenched, and there is little to no value in giving them access to the content. I have data... 7d from a single platform with about 30 forums on this instance. 4.8M hits from Claude 390k from Amazon 261k from Data For SEO 148k from Chat GPT That Claude one! Wowser. Bots that match this (which is also the list I block on some other forums that are fully private by default): (?i). (AhrefsBot|AI…

Would you consider giving these crawlers access if they paid you?

Re: AI companies cause most of traffic on forums

#123

Post by pogue »

equestria wrote: post #24
Ukv wrote: post #9

Are these IPs actually from OpenAI/etc. ( https://openai.com/gptbot.json ), or is it possibly something else masquerading as these bots? The real GPTBot/Amazonbot/etc. claim to obey robots.txt, and switching to a non-bot UA string seems extra questionable behaviour.

I exclude all the published LLM User-Agents and have a content honeypot on my website. Google obeys, but ChatGPT and Bing still clearly know the content of the honeypot.

What's the purpose of the honeypot? Poisoning the LLM or identifying useragents/IPs that shouldn't be seeing it?

Re: AI companies cause most of traffic on forums

#124

Post by jhull »

> And the best thing of all: they crawl the stupidest pages possible. Recently, both ChatGPT and Amazon were - at the same time - crawling the entire edit history of the wiki. And I mean that - they indexed every single diff on every page for every change ever made.

Is it stupid? It makes sense to scrape all these pages and learn the edits and corrections that people make.

Re: AI companies cause most of traffic on forums

#125

Post by echelon »

betaby wrote: post #58
CoastalCoder wrote: post #48

Earlier quoted context omitted.

I wonder if it would work to send Meta's legal department a notice that they are not permitted to access your website. Would that make subsequent accesses be violations of the U.S.'s Computer Fraud and Abuse Act?

Crashing wasn't the intent. And scraping is legal, as I remember per Linkedin case.

Then you can feed them deliberately poisoned data.

Send all of your pages through an adversarial LLM to pollute and twist the meaning of the underlying data.

Re: AI companies cause most of traffic on forums

#126

Post by franga2000 »

betaby wrote: post #58
CoastalCoder wrote: post #48

Earlier quoted context omitted.

I wonder if it would work to send Meta's legal department a notice that they are not permitted to access your website. Would that make subsequent accesses be violations of the U.S.'s Computer Fraud and Abuse Act?

Crashing wasn't the intent. And scraping is legal, as I remember per Linkedin case.

If I make a physical robot and it runs someone over, I'm still liable, even though it was a delivery robot, not a running over people robot.

If a bot sends so many requests that a site completely collapses, the owner is liable, even though it was a scraping bot and not a denial of service bot.

Re: AI companies cause most of traffic on forums

#127

Post by vunderba »

buro9 wrote: post #56

Their appetite cannot be quenched, and there is little to no value in giving them access to the content. I have data... 7d from a single platform with about 30 forums on this instance. 4.8M hits from Claude 390k from Amazon 261k from Data For SEO 148k from Chat GPT That Claude one! Wowser. Bots that match this (which is also the list I block on some other forums that are fully private by default): (?i). (AhrefsBot|AI…

There's also popular repository that maintains a comprehensive list of LLM and AI related bots to aid in blocking these abusive strip miners.

https://github.com/ai-robots-txt/ai.robots.txt

Re: AI companies cause most of traffic on forums

#128

Post by knuppar »

walterbell wrote: post #81
rickyhatespeas wrote: post #68

Earlier quoted context omitted.

It will do harm to their own site considering it's now un-indexable on platforms used by hundreds of millions and growing. Anyone using this is just guaranteeing that their content will be lost to history at worst, or just inaccessible to most search engines/users at best. Congrats on beating the robots, now every time someone searches for your site they will be taken straight to competitors.

> now every time someone searches for your site they will be taken straight to competitors There are non-LLM forms of distribution, including traditional web search and human word of mouth. For some niche websites, a reduction in LLM-search users could be considered a positive community filter. If LLM scraper bots agree to follow longstanding robots.txt protocols, they can join the community of civilized internet par…

Exactly. Not every website needs to be at the top of SEO (or LLM-O?). Increasingly the niche web feels nicer and nicer as centralized platforms expand.

Re: AI companies cause most of traffic on forums

#129

Post by echelon »

bodantogat wrote: post #89
markerz wrote: post #23

One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…

I see a lot of traffic I can tell are bots based on the URL patterns they access. They do not include the "bot" user agent, and often use residential IP pools. I haven't found an easy way to block them. They nearly took out my site a few days ago too.

You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy.

Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills.

Maybe you don't even need a full LLM. Just a simple transformer that inverts negative and positive statements, changes nouns such as locations, and subtly nudges the content into an erroneous state.

Re: AI companies cause most of traffic on forums

#130

Post by buro9 »

ai-christianson wrote: post #122
buro9 wrote: post #56

Their appetite cannot be quenched, and there is little to no value in giving them access to the content. I have data... 7d from a single platform with about 30 forums on this instance. 4.8M hits from Claude 390k from Amazon 261k from Data For SEO 148k from Chat GPT That Claude one! Wowser. Bots that match this (which is also the list I block on some other forums that are fully private by default): (?i). (AhrefsBot|AI…

Would you consider giving these crawlers access if they paid you?

At this point, no.
Post reply on HN