Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

Post reply on HN
Print view View original

251–260 of 466 posts

Re: AI companies cause most of traffic on forums

#251

Post by buro9 »

iLoveOncall wrote: post #244
buro9 wrote: post #56

Their appetite cannot be quenched, and there is little to no value in giving them access to the content. I have data... 7d from a single platform with about 30 forums on this instance. 4.8M hits from Claude 390k from Amazon 261k from Data For SEO 148k from Chat GPT That Claude one! Wowser. Bots that match this (which is also the list I block on some other forums that are fully private by default): (?i). (AhrefsBot|AI…

4.8M requests sounds huge, but if it's over 7 days and especially split amongst 30 websites, it's only a TPS of 0.26, not exactly very high or even abusive. The fact that you choose to host 30 websites on the same instance is irrelevant, those AI bots scan websites, not servers. This has been a recurring pattern I've seen in people complaining about AI bots crawling their website: huge number of requests but actually…

It's never that smooth.

In fact 2M requests arrived on December 23rd from Claude alone for a single site.

Average 25qps is definitely an issue, these are all long tail dynamic pages.

Re: AI companies cause most of traffic on forums

#252

Post by JohnMakin »

MetaWhirledPeas wrote: post #114
markerz wrote: post #23

One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…

> Cloudflare also has a feature to block known AI bots and even suspected AI bots In addition to other crushing internet risks, add wrongly blacklisted as a bot to the list.

These features are opt-in and often paid features. I struggle to see how this is a "crushing risk," although I don't doubt that sufficiently unskilled shops would be completely crushed by an IP/userAgent block. Since Cloudflare has a much more informed and broader view of internet traffic than maybe any other company in the world, I'll probably use that feature without any qualms at some point in the future. Right now their normal WAF rules do a pretty good job of not blocking legitimate traffic, at least on enterprise.

Re: AI companies cause most of traffic on forums

#253

Post by viraptor »

gs17 wrote: post #156
nabla9 wrote: post #108

Earlier quoted context omitted.

The linked article explains what happens when you block their IP.

For reference: > If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). It's really absurd that they seem to think this is acceptable.

Block the whole ASN in that case.

Re: AI companies cause most of traffic on forums

#254

Post by andrethegiant »

CommonCrawl is supposed to help for this, i.e. crawl once and host the dataset for any interested party to download out of band. However, data can be up to a month stale, and it costs $$ to move the data out of us-east-1.

I’m working on a centralized crawling platform[1] that aims to reduce OP’s problem. A caching layer with ~24h TTL for unauthed content would shield websites from redundant bot traffic while still providing up-to-date content for AI crawlers.

[1] https://crawlspace.dev

Re: AI companies cause most of traffic on forums

#255

Post by atomt »

devit wrote: post #88
jsheard wrote: post #78

Earlier quoted context omitted.

That's right, getting DDOSed is a skill issue. Just have infinite capacity.

DDOS is different from crashing. And I doubt Facebook implemented something that actually saturates the network, usually a scraper implements a limit on concurrent connections and often also a delay between connections (e.g. max 10 concurrent, 100ms delay). Chances are the website operator implemented a webserver with terrible RAM efficiency that runs out of RAM and crashes after 10 concurrent requests, or that satur…

I've seen concurrency in excess of 500 from Metas crawlers to a single site. That site had just moved all their images so all the requests hit the "pretty url" rewrite into a slow dynamic request handler. It did not go very well.

Re: AI companies cause most of traffic on forums

#256

Post by alphan0n »

denschub wrote: post #208
alphan0n wrote: post #83

Can someone point out the authors robots.txt where the offense is taking place? I’m just seeing: https://pod.geraspora.de/robots.txt Which allows all user agents. *The discourse server does not disallow the offending bots mentioned in their post: https://discourse.diasporafoundation.org/robots.txt Nor does the wiki: https://wiki.diasporafoundation.org/robots.txt No robots.txt at all on the homepage: https://diasporaf…

the robots.txt on the wiki is no longer what it was when the bot accessed it. primarily because I clean up my stuff afterwards, and the history is now completely inaccessible to non-authenticated users, so there's no need to maintain my custom robots.txt.

https://web.archive.org/web/20240101000000*/https://wiki.dia...

Re: AI companies cause most of traffic on forums

#258

Post by denschub »

alphan0n wrote: post #256
denschub wrote: post #208

Earlier quoted context omitted.

the robots.txt on the wiki is no longer what it was when the bot accessed it. primarily because I clean up my stuff afterwards, and the history is now completely inaccessible to non-authenticated users, so there's no need to maintain my custom robots.txt.

https://web.archive.org/web/20240101000000*/https://wiki.dia...

notice how there's a period of almost two months with no new index, just until a week before I posted this? I wonder what might have caused this!!1

(and it's not like they only check robots.txt once a month or so. https://stuff.overengineer.dev/stash/2024-12-30-dfwiki-opena...)

Re: AI companies cause most of traffic on forums

#259

Post by preommr »

ksec wrote: post #87

On a side note. CGTalk ( One of the largest forum for 3D / VFX Content creation ) was shut down earlier this year citing resources reasons. And for whatever reason they didn't allow anyone to take a copy of the data, run the forum. They could have sold that Data to OpenAI. I wonder how many Forums shut down due to traffics like this? Most of the reason why forums moved to Slack, Discord etc was that they no longer ha…

Every day I get older, and things just get worse. I remember being a young 3d enthusiast trying out blender, game dev etc, and finding resources there. Sad to see that it got shut down.

At least polycount seems to still be around.

Re: AI companies cause most of traffic on forums

#260

Post by ndriscoll »

smolder wrote: post #237
ndriscoll wrote: post #229

Earlier quoted context omitted.

Beyond something like running an ML model, what web pages are expensive (enough that 1-10 requests/second matters at all) to generate these days?

Usually ones that are written in a slow language, do lots of IO to other webservices or databases in a serial, blocking fashion, maybe don't have proper structure or indices in their DBs, and so on. I have seen some really terribly performing spaghetti web sites, and have experience with them collapsing under scraping load. With a mountain of technical debt in the way it can even be challenging to fix such a thing.

Even if you're doing serial IO on a single thread, I'd expect you should be able to handle hundreds of qps. I'd think a slow language wouldn't be 1000x slower than something like functional scala. It could be slow if you're missing an index, but then I'd expect the thing to barely run for normal users; scraping at 2/s isn't really the issue there.
Post reply on HN