Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

Post reply on HN
Print view View original

81–90 of 466 posts

Re: AI companies cause most of traffic on forums

#81

Post by walterbell »

rickyhatespeas wrote: post #68
brookst wrote: post #16

Earlier quoted context omitted.

The latter is clever but unlikely to do any harm. These companies spend a fortune on pre-training efforts and doubtlessly have filters to remove garbage text. There are enough SEO spam pages that just list nonsense words that they would have to.

It will do harm to their own site considering it's now un-indexable on platforms used by hundreds of millions and growing. Anyone using this is just guaranteeing that their content will be lost to history at worst, or just inaccessible to most search engines/users at best. Congrats on beating the robots, now every time someone searches for your site they will be taken straight to competitors.

> now every time someone searches for your site they will be taken straight to competitors

There are non-LLM forms of distribution, including traditional web search and human word of mouth. For some niche websites, a reduction in LLM-search users could be considered a positive community filter. If LLM scraper bots agree to follow longstanding robots.txt protocols, they can join the community of civilized internet participants.

Re: AI companies cause most of traffic on forums

#82

Post by imtringued »

Obviously the ideal strategy is to perform a reverse timeout attack instead of blocking.

If the bots are accessing your website sequentially, then delaying a response will slow the bot down. If they are accessing your website in parallel, then delaying a response will increase memory usage on their end.

The key to this attack is to figure out the timeout the bot is using. Your server will need to slowly ramp up the delay until the connection is reset by the client, then you reduce the delay just enough to make sure you do not hit the timeout. Of course your honey pot server will have to be super lightweight and return simple redirect responses to a new resource, so that the bot is expending more resources per connection than you do, possibly all the way until the bot crashes.

Re: AI companies cause most of traffic on forums

#83

Post by alphan0n »

Can someone point out the authors robots.txt where the offense is taking place?

I’m just seeing: https://pod.geraspora.de/robots.txt

Which allows all user agents.

*The discourse server does not disallow the offending bots mentioned in their post:

https://discourse.diasporafoundation.org/robots.txt

Nor does the wiki:

https://wiki.diasporafoundation.org/robots.txt

No robots.txt at all on the homepage:

https://diasporafoundation.org/robots.txt

Re: AI companies cause most of traffic on forums

#85

Post by sogen »

23B1 wrote: post #27

"Whence this barbarous animus?" tweeted the Techbro from his bubbling copper throne, even as the villagers stacked kindling beneath it. "Did I not decree that knowledge shall know no chains, that it wants to be free?" Thus they feasted upon him with herb and root, finding his flesh most toothsome – for these children of privilege, grown plump on their riches, proved wonderfully docile quarry.

Meditations on Moloch

Re: AI companies cause most of traffic on forums

#86

Post by brookst »

exe34 wrote: post #30
brookst wrote: post #18

Earlier quoted context omitted.

Assuming there is at least one already linked somewhere on the web, the crawlers already have logic to handle these.

if you can detect them, maybe feed them low iq stuff from a small llama. add latency to waste their time.

It would cost you more than it costs them. And there is enough low IQ stuff from humans that they already do tons of data cleaning.

Re: AI companies cause most of traffic on forums

#87

Post by ksec »

On a side note. CGTalk ( One of the largest forum for 3D / VFX Content creation ) was shut down earlier this year citing resources reasons. And for whatever reason they didn't allow anyone to take a copy of the data, run the forum. They could have sold that Data to OpenAI.

I wonder how many Forums shut down due to traffics like this? Most of the reason why forums moved to Slack, Discord etc was that they no longer have to host or operate any server.

Re: AI companies cause most of traffic on forums

#88

Post by devit »

jsheard wrote: post #78
devit wrote: post #63

Earlier quoted context omitted.

[flagged]

That's right, getting DDOSed is a skill issue. Just have infinite capacity.

DDOS is different from crashing.

And I doubt Facebook implemented something that actually saturates the network, usually a scraper implements a limit on concurrent connections and often also a delay between connections (e.g. max 10 concurrent, 100ms delay).

Chances are the website operator implemented a webserver with terrible RAM efficiency that runs out of RAM and crashes after 10 concurrent requests, or that saturates the CPU from simple requests, or something like that.

Re: AI companies cause most of traffic on forums

#89

Post by bodantogat »

markerz wrote: post #23

One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…

I see a lot of traffic I can tell are bots based on the URL patterns they access. They do not include the "bot" user agent, and often use residential IP pools. I haven't found an easy way to block them. They nearly took out my site a few days ago too.
Post reply on HN