AI companies cause most of traffic on forums • Hacker News Board

Re: AI companies cause most of traffic on forums

#241

Post by viraptor » Mon, Dec 30, 2024, 7:53 PM UTC

markerz wrote: post #23

One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…

You can also block by IP. Facebook traffic comes from a single ASN and you can kill it all in one go, even before user agent is known. The only thing this potentially affects that I know of is getting the social card for your site.

Re: AI companies cause most of traffic on forums

#242

Post by smolder » Mon, Dec 30, 2024, 7:53 PM UTC

frereubu wrote: post #80

coldpie wrote: post #65

Earlier quoted context omitted.

You know, at this point, I wonder if an allowlist would work better.

I love (hate) the idea of a site where you need to send a personal email to the webmaster to be whitelisted.

We just need a browser plugin to auto-email webmasters to request access, and wait for the follow-up "access granted" email. It could be powered by AI.

Re: AI companies cause most of traffic on forums

#243

Post by Terr_ » Mon, Dec 30, 2024, 7:54 PM UTC

coldpie wrote: post #215

qazxcvbnmlp wrote: post #200

Earlier quoted context omitted.

Everything on this can be programmatically simulated by a bot with bad intentions. It will be a cat and mouse game of finding behaviors that differentiate between bot and not and patching them. To truly say “I trust real browsers” requires a signal of integrity of the user and browser such as cryptographic device attestation of the browser. .. which has to be centrally verified. Which is also not great.

> Everything on this can be programmatically simulated by a bot with bad intentions. It will be a cat and mouse game of finding behaviors that differentiate between bot and not and patching them. Forcing Facebook & Co to play the adversary role still seems like an improvement over the current situation. They're clearly operating illegitimately if they start spoofing real user agents to get around bot blocking capabil…

I'm imagining a quixotic terms of service, where "by continuing" any bot access grants the site-owner a perpetual and irrevocable license to use and relicense all data, works, or other products resulting from any use of the crawled content, including but not limited to cases where that content was used in a statistical text generative model.

Re: AI companies cause most of traffic on forums

#244

Post by iLoveOncall » Mon, Dec 30, 2024, 7:55 PM UTC

buro9 wrote: post #56

Their appetite cannot be quenched, and there is little to no value in giving them access to the content. I have data... 7d from a single platform with about 30 forums on this instance. 4.8M hits from Claude 390k from Amazon 261k from Data For SEO 148k from Chat GPT That Claude one! Wowser. Bots that match this (which is also the list I block on some other forums that are fully private by default): (?i). (AhrefsBot|AI…

4.8M requests sounds huge, but if it's over 7 days and especially split amongst 30 websites, it's only a TPS of 0.26, not exactly very high or even abusive.

The fact that you choose to host 30 websites on the same instance is irrelevant, those AI bots scan websites, not servers.

This has been a recurring pattern I've seen in people complaining about AI bots crawling their website: huge number of requests but actually a low TPS once you dive a bit deeper.

Re: AI companies cause most of traffic on forums

#245

Post by cess11 » Mon, Dec 30, 2024, 7:55 PM UTC

echelon wrote: post #125

betaby wrote: post #58

Earlier quoted context omitted.

Crashing wasn't the intent. And scraping is legal, as I remember per Linkedin case.

Then you can feed them deliberately poisoned data. Send all of your pages through an adversarial LLM to pollute and twist the meaning of the underlying data.

The scraper bots can remain irrational longer than you can stay solvent.

Re: AI companies cause most of traffic on forums

#246

Post by raphman » Mon, Dec 30, 2024, 7:56 PM UTC

IanKerr wrote: post #164

mentalgear wrote: post #66

Note-worthy from the article (as some commentators suggested blocking them). "If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet."

This is the beginning of the end of the public internet, imo. Websites that aren't able to manage the bandwidth consumption of AI scrapers and the endless spam that will take over from LLMs writing comments on forums are going to go under. The only things left after AI has its way will be walled gardens with whitelisted entrants or communities on large websites like Facebook. Niche, public sites are going to become u…

Yeah. Our research group has a wiki with (among other stuff) a list of open, completed, and ongoing bachelor's/master's theses. Until recently, the list was openly available. But AI bots caused significant load by crawling each page hundreds of times, following all links to tags (which are implemented as dynamic searches), prior revisions, etc. Since a few weeks, the pages are only available to authenticated users.

Re: AI companies cause most of traffic on forums

#247

Post by iLoveOncall » Mon, Dec 30, 2024, 7:57 PM UTC

> That equals to 2.19 req/s - which honestly isn't that much

This is the only thing that matters.

Re: AI companies cause most of traffic on forums

#248

Post by benhurmarcel » Mon, Dec 30, 2024, 8:00 PM UTC

throwaway290 wrote: post #119

MetaWhirledPeas wrote: post #114

Earlier quoted context omitted.

> Cloudflare also has a feature to block known AI bots and even suspected AI bots In addition to other crushing internet risks, add wrongly blacklisted as a bot to the list.

What do you mean crushing risk? Just solve these 12 puzzles by moving tiny icons on tiny canvas while on the phone and you are in the clear for a couple more hours!

Sometimes it doesn’t even give you a Captcha.

I have come across some websites that block me using Cloudflare with no way of solving it. I’m not sure why, I’m in a large first-world country, I tried a stock iPhone and a stock Windows PC, no VPN or anything.

That’s just no way to know.

Re: AI companies cause most of traffic on forums

#249

Post by oehpr » Mon, Dec 30, 2024, 8:04 PM UTC

PaulRobinson wrote: post #162

If they're not respecting robots.txt, and they're causing degradation in service, it's unauthorised access, and therefore arguably criminal behaviour in multiple jurisdictions. Honestly, call your local cyber-interested law enforcement. NCSC in UK, maybe FBI in US? Genuinely, they'll not like this. It's bad enough that we have DDoS from actual bad actors going on, we don't need this as well.

It's honestly depressing.

Any normal human would be sued into complete oblivion over this. But everyone knows that these laws arn't meant to be used against companies like this. Only us. Only ever us.

Re: AI companies cause most of traffic on forums

#250

Post by pacifika » Mon, Dec 30, 2024, 8:04 PM UTC

mentalgear wrote: post #66

Note-worthy from the article (as some commentators suggested blocking them). "If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet."

So you get all the IPs by rate limiting them?