Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

Post reply on HN
Print view View original

421–430 of 466 posts

Re: AI companies cause most of traffic on forums

#421

Post by lazide »

brianwawok wrote: post #406
shark_laser wrote: post #348

Earlier quoted context omitted.

Yep. Same for most of Asia too. Cloudflare's filters are basically straight up racist. I have stopped using so many sites due to their use of Cloudflare.

If 90% of your problem users come from 1-2 countries, seems pretty sensible to block that country. I know I have 0 paying users in those countries, so why deal with it? Let them go fight it out doing bot wars in local sites

Keep in mind, this is literally why stereotypes and racism exists. It’s the exact same process/reasoning.

Re: AI companies cause most of traffic on forums

#422

Post by d0mine »

crote wrote: post #409
d0mine wrote: post #391

Earlier quoted context omitted.

unrelated: USSR might have been 2nd world. Russia is 3rd world (since 1991) -- banana republic

No, Russia is by definition the 2nd world. It's about spheres of influence, not any kind of economic status. The First World is the Western Bloc centered around the US, the Second World is the Eastern Bloc centered around then-USSR and now-Russia (although these days more centered on China), the Third World is everyone else.

By which definition? Here’s the first result in google: “The term "second world" was initially used to refer to the Soviet Union and countries of the communist bloc. It has subsequently been revised to refer to nations that fall between first and third world countries in terms of their development status and economic indicators.” https://www.investopedia.com/terms/s/second-world.asp#:~:tex....

Notice the word economic in it.

Re: AI companies cause most of traffic on forums

#423

Post by tonyedgecombe »

candlemas wrote: post #224
markerz wrote: post #23

One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…

The biggest offenders for my website have always been from China.

[flagged]

Re: AI companies cause most of traffic on forums

#424

Post by victorbjorklund »

panic wrote: post #334
ls612 wrote: post #324

Earlier quoted context omitted.

People hate collective punishment because it works so well.

Works how? Are these blocks leading to progress toward solving any of the underlying issues?

The underlying issue is that countries like russia support abuse like this. So by blocking them perhaps the people there will demand that their govt stops supporting crimes and absuse so that they can be allowed back into the internet.

(In the case of russians though i guess they will never change)

Re: AI companies cause most of traffic on forums

#425

Post by odo1242 »

datadrivenangel wrote: post #230
bodantogat wrote: post #188

Earlier quoted context omitted.

I have no way to verify this, I suspect these are either stealth AI companies or data collectors, who hope to sell training data to them

I've heard that some mobile SDKs / Apps earn extra revenue by providing an IP address for VPN connections / scraping.

Chrome extensions too

Re: AI companies cause most of traffic on forums

#426

Post by [deleted] »

eckesicle wrote: post #328
ls612 wrote: post #324

Earlier quoted context omitted.

People hate collective punishment because it works so well.

Anecdatally, by default, we now block all Chinese and Russian IPs across our servers. After doing so, all of our logs, like ssh auth etc, are almost completely free and empty of malicious traffic. It’s actually shocking how well a blanket ban worked for us.

[deleted]

Re: AI companies cause most of traffic on forums

#427

Post by petre »

victorbjorklund wrote: post #424
panic wrote: post #334

Earlier quoted context omitted.

Works how? Are these blocks leading to progress toward solving any of the underlying issues?

The underlying issue is that countries like russia support abuse like this. So by blocking them perhaps the people there will demand that their govt stops supporting crimes and absuse so that they can be allowed back into the internet. (In the case of russians though i guess they will never change)

> people there will demand that their govt stops supporting crimes and absuse so that they can be allowed back into the internet

Sure. It doesn't work that way, not in Russia or China. First they have to revert back to 1999 when Putin took over. Then they have to extradite criminals and crack down on cybercrime. Then maybe they could be allowed back onto the open Internet.

In my country one would be exradited to the US in no time. In fact the USSS came over for a guy who had been laundering money through BTC from a nearby office. Not a month passed and he got extradited to the US, never to be heard from again.

Re: AI companies cause most of traffic on forums

#428

Post by petre »

bodantogat wrote: post #89
markerz wrote: post #23

One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…

I see a lot of traffic I can tell are bots based on the URL patterns they access. They do not include the "bot" user agent, and often use residential IP pools. I haven't found an easy way to block them. They nearly took out my site a few days ago too.

You rate limit them and then block the abusers. Nginx allows rate limiting. You can then block them using fail2ban for an hour if they're rate limited 3 times. If they get blocked 5 times you can block them forever using the recidive jail.

I've had massive AI bot traffic from M$, blocked several IPs by adding manual entries into the recidive jail. If they come back and disregard robots.txt with disallow * I will run 'em through fail2ban.

Re: AI companies cause most of traffic on forums

#430

Post by shkkmo »

alphan0n wrote: post #400
ribadeo wrote: post #389

Earlier quoted context omitted.

How do you know what the contextual configuration of their robots.txt is/was? Your accusation was directly addressed by the author in a comment on the original post, IIRC i find your attitude as expressed here to be problematic in many ways

CommonCrawl archives robots.txt For convenience, you can view the extracted data here: https://pastebin.com/VSHMTThJ You are welcome to verify for yourself by searching for “wiki.diasporafoundation.org/robots.txt” in the CommonCrawl index here: https://index.commoncrawl.org/ The index contains a file name that you can append to the CommonCrawl url to download the archive and view. More detailed information on downloa…

Your attitude is inappropriate and violates the sitewide guidelines for discussion.
Post reply on HN