Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

Post reply on HN
Print view View original

311–320 of 466 posts

Re: AI companies cause most of traffic on forums

#311

Post by neilv »

MetaWhirledPeas wrote: post #273
JohnMakin wrote: post #252

Earlier quoted context omitted.

These features are opt-in and often paid features. I struggle to see how this is a "crushing risk," although I don't doubt that sufficiently unskilled shops would be completely crushed by an IP/userAgent block. Since Cloudflare has a much more informed and broader view of internet traffic than maybe any other company in the world, I'll probably use that feature without any qualms at some point in the future. Right no…

The risk is not to the company using Cloudflare; the risk is to any legitimate individual who Cloudflare decides is a bot. Hopefully their detection is accurate because a false positive would cause great difficulties for the individual.

For months, my Firefox was locked out of gitlab.com and some other sites I wanted to use, because CloudFlare didn't like my browser.

Lesson learned: even when you contact the sales dept. of multiple companies, they just don't/can't care about random individuals.

Even if they did care, a company successfully doing an extended three-way back-and-forth troubleshooting with CloudFlare, over one random individual, seems unlikely.

Re: AI companies cause most of traffic on forums

#312

Post by wood_spirit »

brookst wrote: post #16
walterbell wrote: post #10

OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scraper... For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/ > The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the c…

The latter is clever but unlikely to do any harm. These companies spend a fortune on pre-training efforts and doubtlessly have filters to remove garbage text. There are enough SEO spam pages that just list nonsense words that they would have to.

Rather than garbage, perhaps just serve up something irrelevant and banal? Or splice sentences from various random project Gutenberg books? And add in a tarpit for good measure.

At least in the end it gives the programmer one last hoorah before the AI makes us irrelevant :)

Re: AI companies cause most of traffic on forums

#313

Post by IncreasePosts »

Capricorn2481 wrote: post #305
kmoser wrote: post #286

Earlier quoted context omitted.

My cheap and dirty way of dealing with bots like that is to block any IP address that accesses any URLs in robots.txt. It's not a perfect strategy but it gives me pretty good results given the simplicity to implement.

I don't understand this. You don't have routes your users might need in robots.txt? This article is about bots accessing resources that other might use.

It seems better to put fake honeypot urls in robots.txt, and block any up that accesses those.

Re: AI companies cause most of traffic on forums

#314

Post by lolinder »

tecoholic wrote: post #310

For any self-hosting enthusiasts out here. Check your network traffic if you have a Gitea instance running. My network traffic was mostly just AmazonBot and some others from China hitting every possible URL constantly. My traffic has gone from 2-5GB per day to a tenth of that after blocking the bots.

This is one of many reasons why I don't host on the open internet. All my stuff is running on my local network, accessible via VPN if needed.

Re: AI companies cause most of traffic on forums

#316

Post by jgalt212 »

imtringued wrote: post #82

Obviously the ideal strategy is to perform a reverse timeout attack instead of blocking. If the bots are accessing your website sequentially, then delaying a response will slow the bot down. If they are accessing your website in parallel, then delaying a response will increase memory usage on their end. The key to this attack is to figure out the timeout the bot is using. Your server will need to slowly ramp up the d…

> delaying a response will slow the bot down

This is a nice solution for an asynchronous web server. For apache, not so much.

Re: AI companies cause most of traffic on forums

#317

Post by vultour »

l1n wrote: post #280
buro9 wrote: post #99

Earlier quoted context omitted.

Nginx, it's nothing special it's just my load balancer. if ($http_user_agent ~* (list|of|case|insensitive|things|to|block)) {return 403;}

403 is generally a bad way to get crawlers to go away - https://developers.google.com/search/blog/2023/02/dont-404-m... suggests a 500, 503, or 429 HTTP status code.

That article describes the exact behaviour you want from the AI crawlers. If you let them know they’re rate limited they’ll just change IP or user agent.

Re: AI companies cause most of traffic on forums

#318

Post by Dilettante_ »

Mistletoe wrote: post #107
buro9 wrote: post #56

Their appetite cannot be quenched, and there is little to no value in giving them access to the content. I have data... 7d from a single platform with about 30 forums on this instance. 4.8M hits from Claude 390k from Amazon 261k from Data For SEO 148k from Chat GPT That Claude one! Wowser. Bots that match this (which is also the list I block on some other forums that are fully private by default): (?i). (AhrefsBot|AI…

This is a new twist on the Dead Internet Theory I hadn’t thought of.

We'll have two entirely separate (dead) internets! One for real hosts who will only get machine users, and one for real users who only get machine content!

Wait, that seems disturbingly conceivable with the way things are going right now. *shudder*

Re: AI companies cause most of traffic on forums

#319

Post by Dilettante_ »

mtnGoat wrote: post #34

Some of these ai companies are so aggressive they are essentially dos’ing sites offline with their request volumes. Should be careful before they get blacked and can’t get data anymore. ;)

>before they get blacked

...Please don't phrase it like that.

Re: AI companies cause most of traffic on forums

#320

Post by Capricorn2481 »

IncreasePosts wrote: post #313
Capricorn2481 wrote: post #305

Earlier quoted context omitted.

I don't understand this. You don't have routes your users might need in robots.txt? This article is about bots accessing resources that other might use.

It seems better to put fake honeypot urls in robots.txt, and block any up that accesses those.

Ah I see
Post reply on HN