Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

Post reply on HN
Print view View original

441–450 of 466 posts

Re: AI companies cause most of traffic on forums

#441

Post by herbst »

azemetre wrote: post #70
betaby wrote: post #58

Earlier quoted context omitted.

Crashing wasn't the intent. And scraping is legal, as I remember per Linkedin case.

There’s a fine line between scrapping and DDOS’ing I’m sure. Just because you manufacture chemicals doesn’t mean you can legally dump your toxic waste anywhere you want (well shouldn’t be allowed to at least). You also shouldn’t be able to set your crawlers causing sites to fail.

It's like these AI companies have to invent scraping spiders again from scratch. I don't know how often I have been ddosed to complete site failure and still ongoing by random scrapers just the last few months.

Re: AI companies cause most of traffic on forums

#443

Post by lazide »

qeternity wrote: post #439
lazide wrote: post #421

Earlier quoted context omitted.

Keep in mind, this is literally why stereotypes and racism exists. It’s the exact same process/reasoning.

No, racism would be “I won’t deal with customers of Chinese ethnicity irrespective of their country of operation”. Blocking Chinese (or whatever) IPs because they are responsible for a huge amount of malicious behavior is not racist. Frankly I don’t care what the race of the Chinese IP threat actor is.

You really might want to re-read my comment.

Re: AI companies cause most of traffic on forums

#444

Post by [deleted] »

lazide wrote: post #421
brianwawok wrote: post #406

Earlier quoted context omitted.

If 90% of your problem users come from 1-2 countries, seems pretty sensible to block that country. I know I have 0 paying users in those countries, so why deal with it? Let them go fight it out doing bot wars in local sites

Keep in mind, this is literally why stereotypes and racism exists. It’s the exact same process/reasoning.

[deleted]

Re: AI companies cause most of traffic on forums

#445

Post by [deleted] »

crote wrote: post #409
d0mine wrote: post #391

Earlier quoted context omitted.

unrelated: USSR might have been 2nd world. Russia is 3rd world (since 1991) -- banana republic

No, Russia is by definition the 2nd world. It's about spheres of influence, not any kind of economic status. The First World is the Western Bloc centered around the US, the Second World is the Eastern Bloc centered around then-USSR and now-Russia (although these days more centered on China), the Third World is everyone else.

[deleted]

Re: AI companies cause most of traffic on forums

#446

Post by danlugo92 »

tivert wrote: post #270
echelon wrote: post #129

Earlier quoted context omitted.

You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy. Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills. Maybe you don't even need a full LLM. Just a simple transfor…

> You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy. > Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills. I agree, and not just to discourage them running up traf…

If your employer can run their companies without employees in the future it also means you can have your own company with no employees.

If anything this will level the playing field, and creativity will prevail.

Re: AI companies cause most of traffic on forums

#447

Post by hmottestad »

Saris wrote: post #431

What I don't get is why they need to crawl so aggressively, I have a site with content that doesn't change often (company website) with a few hundred pages total. But the same AI bot will scan the entire site multiple times per day, like somehow all the content is going to suddenly change now after it hasn't for months. That cannot be an efficient use of their money, maybe they used their own AI to write the scraper…

The post mentions that the bots were crawling all the wiki diffs. I think that might be useful to see how text evolves and changes over time. Possibly how it improves over time, and what those improvements are.

I guess they are hoping that there will be small changes to your website that it can learn from.

Re: AI companies cause most of traffic on forums

#448

Post by tivert »

danlugo92 wrote: post #446
tivert wrote: post #270

Earlier quoted context omitted.

> You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy. > Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills. I agree, and not just to discourage them running up traf…

If your employer can run their companies without employees in the future it also means you can have your own company with no employees. If anything this will level the playing field, and creativity will prevail.

> If your employer can run their companies without employees in the future it also means you can have your own company with no employees.

No, you still need money. Lots of money.

> If anything this will level the playing field, and creativity will prevail.

That's a fantasy. The people that already have money will prevail (for the most part).

Re: AI companies cause most of traffic on forums

#450

Post by petre »

herbst wrote: post #440
petre wrote: post #428

Earlier quoted context omitted.

You rate limit them and then block the abusers. Nginx allows rate limiting. You can then block them using fail2ban for an hour if they're rate limited 3 times. If they get blocked 5 times you can block them forever using the recidive jail. I've had massive AI bot traffic from M$, blocked several IPs by adding manual entries into the recidive jail. If they come back and disregard robots.txt with disallow * I will run…

Whatever M$ was doing still baffles me. I still have several azure ranges in my blocklist because whatever this was appeared to change strategie once I implemented a ban method.

They were hammering our closed ticketing system for some reason. I blocked an entire C block and an individual IP. If needed I will not hesitate banning all their ranges, which means we won't get any mail from Azure, M$ office 365, since this is also our mail server. But scew'em, I'll do it anyway until someone notices, since it's clearly abuse.
Post reply on HN