Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

Post reply on HN
Print view View original

11–20 of 466 posts

Re: AI companies cause most of traffic on forums

#11

Post by Kapura »

varelse wrote: post #6
Kapura wrote: post #3

Earlier quoted context omitted.

[flagged]

It's certainly one of the few things that actually gets their attention. But aren't there more important things than this for the Luigis among us? I would suspect there's good money in offering a service to detect AI content on all of these forums and reject it. That will then be used as training data to refine them which gives such a service infinite sustainability.

>I would suspect there's good money in offering a service to detect AI content on all of these forums and reject it

This sounds like the cheater/anti-cheat arms race in online multiplayer games. Cheat developers create something, the anti-cheat teams create a method to detect and reject the exploit, a new cheat is developed, and the cycle continues. But this is much lower stakes than AI trying to vacuum up all of human expression, or trick real humans into wasting their time talking to computers.

Re: AI companies cause most of traffic on forums

#14

Post by stainablesteel »

johng wrote: post #2

If they ignore robots.txt there should be some kind of recourse :(

court ruling a few years ago said it's legal to scrape web pages, you don't need to be respectful of these for any purely legal reasons

however this doesn't stop the website from doing what they can to stop scraping attempts, or using a service to do that for them

Re: AI companies cause most of traffic on forums

#15

Post by BryantD »

I can understand why LLM companies might want to crawl those diffs -- it's context. Assuming that we've trained LLM on all the low hanging fruit, building a training corpus that incorporates the way a piece of text changes over time probably has some value. This doesn't excuse the behavior, of course.

Back in the day, Google published the sitemap protocol to alleviate some crawling issues. But if I recall correctly, that was more about helping the crawlers find more content, not controlling the impact of the crawlers on websites.

Re: AI companies cause most of traffic on forums

#16

Post by brookst »

walterbell wrote: post #10

OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scraper... For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/ > The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the c…

The latter is clever but unlikely to do any harm. These companies spend a fortune on pre-training efforts and doubtlessly have filters to remove garbage text. There are enough SEO spam pages that just list nonsense words that they would have to.

Re: AI companies cause most of traffic on forums

#17

Post by jsheard »

BryantD wrote: post #15

I can understand why LLM companies might want to crawl those diffs -- it's context. Assuming that we've trained LLM on all the low hanging fruit, building a training corpus that incorporates the way a piece of text changes over time probably has some value. This doesn't excuse the behavior, of course. Back in the day, Google published the sitemap protocol to alleviate some crawling issues. But if I recall correctly,…

The sitemap protocol does have some features to help avoid unnecessary crawling, you can specify the last time each page was modified and roughly how frequently they're expected to be modified in the future so that crawlers can skip pulling them again when nothing has meaningfully changed.

Re: AI companies cause most of traffic on forums

#19

Post by yodsanklai »

stainablesteel wrote: post #14
johng wrote: post #2

If they ignore robots.txt there should be some kind of recourse :(

court ruling a few years ago said it's legal to scrape web pages, you don't need to be respectful of these for any purely legal reasons however this doesn't stop the website from doing what they can to stop scraping attempts, or using a service to do that for them

> court ruling

Isn't this country dependent though?

Re: AI companies cause most of traffic on forums

#20

Post by walterbell »

brookst wrote: post #16
walterbell wrote: post #10

OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scraper... For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/ > The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the c…

The latter is clever but unlikely to do any harm. These companies spend a fortune on pre-training efforts and doubtlessly have filters to remove garbage text. There are enough SEO spam pages that just list nonsense words that they would have to.

Obfuscators can evolve alongside other LLM arms races.
Post reply on HN