Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

Post reply on HN
Print view View original

391–400 of 466 posts

Re: AI companies cause most of traffic on forums

#391

Post by d0mine »

kmeisthax wrote: post #322
MetaWhirledPeas wrote: post #114

Earlier quoted context omitted.

> Cloudflare also has a feature to block known AI bots and even suspected AI bots In addition to other crushing internet risks, add wrongly blacklisted as a bot to the list.

This is already a thing for basically all of the second[0] and third worlds. A non-trivial amount of Cloudflare's security value is plausible algorithmic discrimination and collective punishment as a service. [0] Previously Soviet-aligned countries; i.e. Russia and eastern Europe.

unrelated: USSR might have been 2nd world. Russia is 3rd world (since 1991) -- banana republic

Re: AI companies cause most of traffic on forums

#392

Post by TacticalCoder »

eckesicle wrote: post #328
ls612 wrote: post #324

Earlier quoted context omitted.

People hate collective punishment because it works so well.

Anecdatally, by default, we now block all Chinese and Russian IPs across our servers. After doing so, all of our logs, like ssh auth etc, are almost completely free and empty of malicious traffic. It’s actually shocking how well a blanket ban worked for us.

> Anecdatally, by default, we now block all Chinese and Russian IPs across our servers.

This. Just get several countries' entire IP address space and block these. I've posted I was doing just that only to be told that this wasn't in the "spirit" of the Internet or whatever similar nonsense.

In addition to that only allow SSH in from the few countries / ISPs legit trafic shall legitimately be coming from. This quiets the logs, saves bandwidth, saves resources, saves the planet.

Re: AI companies cause most of traffic on forums

#393

Post by oriettaxx »

last week we had to double AWS-RDS database CPU, ... and the biggest load was from AmazonBot:

the weird is:

1. AmazonBot traffic imply we give more money to AWS (in terms of CPU, DB cpu, and traffic, too)

2. What the hell is AmazonBot doing? what's the point of that crawler?

Re: AI companies cause most of traffic on forums

#395

Post by misswaterfairy »

yihongs wrote: post #201

Funny thing is half these websites are probably served over cloud so Google, Amazon, and MSFT DDoS themselves and charge the clients for traffic.

Another HN user experiencing this: https://news.ycombinator.com/item?id=42567896

They're stealing their customers data, and they're charging them for the privilege...

Re: AI companies cause most of traffic on forums

#396

Post by misswaterfairy »

cs702 wrote: post #148

AI companies go on forums to scrape content for training models, which are surreptitiously used to generate content posted on forums, from which AI companies scrape content to train models, which are surreptitiously used to generate content posted on forums... It's a lot of traffic, and a lot of new content, most of which seems to add no value. Sigh.

One hopes that this will eventually burst the AI bubble.

Re: AI companies cause most of traffic on forums

#397

Post by aftbit »

layer8 wrote: post #163
aftbit wrote: post #93

Earlier quoted context omitted.

Yeah, this is the sort of thing that a caching and rate limiting load balancer (e.g. nginx) could very trivially mitigate. Just add a request limit bucket based on the meta User Agent allowing at most 1 qps or whatever (tune to 20% of your backend capacity), returning 429 when exceeded. Of course Cloudflare can do all of this for you, and they functionally have unlimited capacity.

Read the article, the bots change their User Agent to an innocuous one when they start being blocked. And having to use Cloudflare is just as bad for the internet as a whole as bots routinely eating up all available resources.

I did read the article. I'm skeptical of the claim though. The author was careful to publish specific UAs for the bots, but then provided no extra information of the non-bot UAs.

>If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet.

I'm also skeptical of the need for _anyone_ to access the edit history at 10 qps. You could put an nginx rule on those routes that just limits the edit history page to 0.5 qps per IP and 2 qps across all IPs, which would protect your site from both bad AI bots and dumb MediaWiki script kiddies at little impact.

>Oh, and of course, they don’t just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not.

And caching would fix this too, especially for pages that are guaranteed not to change (e.g. an edit history diff page).

Don't get me wrong, I'm not unsympathetic to the author's plight, but I do think that the internet is an unsafe place full of bad actors, and a single bad actor can easily cause a lot of harm. I don't think throwing up your arms and complaining is that helpful. Instead, just apply the mitigations that have existed for this for at least 15 years, and move on with your life. Your visitors will be happier and the bots will get boned.

Re: AI companies cause most of traffic on forums

#398

Post by Dilettante_ »

ribadeo wrote: post #390
Dilettante_ wrote: post #319

Earlier quoted context omitted.

>before they get blacked ...Please don't phrase it like that.

Its probably 'blocked' misspelled, given the context. Not everyone speaks English as a first language

Oh that makes more sense. I read it as an unfortunately chosen abbreviation of "blacklisted".

Re: AI companies cause most of traffic on forums

#399

Post by kmoser »

Beijinger wrote: post #333
kmoser wrote: post #286

Earlier quoted context omitted.

My cheap and dirty way of dealing with bots like that is to block any IP address that accesses any URLs in robots.txt. It's not a perfect strategy but it gives me pretty good results given the simplicity to implement.

How can I implement this?

Too many ways to list here, and implementation details will depend on your hosting environment and other requirements. But my quick-and-dirty trick involves a single URL which, when visited, runs a script which appends "deny from foo" (where foo is the naughty IP address) to my .htaccess file. The URL in question is not publicly listed, so nobody will accidentally stumble upon it and accidentally ban themselves. It's also specifically disallowed in robots.txt, so in theory it will only be visited by bad bots.

Re: AI companies cause most of traffic on forums

#400

Post by alphan0n »

ribadeo wrote: post #389
alphan0n wrote: post #284

Earlier quoted context omitted.

I would take anything the author said with a grain of salt. They straight up lied about the configuration of the robots.txt file. https://news.ycombinator.com/item?id=42551628

How do you know what the contextual configuration of their robots.txt is/was? Your accusation was directly addressed by the author in a comment on the original post, IIRC i find your attitude as expressed here to be problematic in many ways

CommonCrawl archives robots.txt

For convenience, you can view the extracted data here:

https://pastebin.com/VSHMTThJ

You are welcome to verify for yourself by searching for “wiki.diasporafoundation.org/robots.txt” in the CommonCrawl index here:

https://index.commoncrawl.org/

The index contains a file name that you can append to the CommonCrawl url to download the archive and view.

More detailed information on downloading archives here:

https://commoncrawl.org/get-started

From September to December, the robots.txt at wiki.diasporafoundation.org contained this, and only this:

>User-agent: * >Disallow: /w/

Apologies for my attitude, I find defenders of the dishonest in the face of clear evidence even more problematic.

Post reply on HN