Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

Post reply on HN
Print view View original

281–290 of 466 posts

Re: AI companies cause most of traffic on forums

#281

Post by ryandrake »

markerz wrote: post #23

One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…

> My solution was to add a Cloudflare rule to block requests from their User-Agent.

Surely if you can block their specific User-Agent, you could also redirect their User-Agent to goatse or something. Give em what they deserve.

Re: AI companies cause most of traffic on forums

#282

Post by l1n »

buro9 wrote: post #251
iLoveOncall wrote: post #244

Earlier quoted context omitted.

4.8M requests sounds huge, but if it's over 7 days and especially split amongst 30 websites, it's only a TPS of 0.26, not exactly very high or even abusive. The fact that you choose to host 30 websites on the same instance is irrelevant, those AI bots scan websites, not servers. This has been a recurring pattern I've seen in people complaining about AI bots crawling their website: huge number of requests but actually…

It's never that smooth. In fact 2M requests arrived on December 23rd from Claude alone for a single site. Average 25qps is definitely an issue, these are all long tail dynamic pages.

Curious what your robots.txt looked like, if you have a link?

Re: AI companies cause most of traffic on forums

#283

Post by Terr_ »

m3047 wrote: post #195
uludag wrote: post #103

I'm always curious how poisoning attacks could work. Like, suppose that you were able to get enough human users to produce poisoned content. This poisoned content would be human written and not just garbage, and would contain flawed reasoning, misjudgments, lapses of reasoning, unrealistic premises, etc. Like, I've asked ChatGPT certain questions where I know the online sources are limited and it would seem that from…

(I was going to post "run a bot motel" as a topline, but I get tired of sounding like broken record.) To generate garbage data I've had good success using Markov Chains in the past. These days I think I'd try an LLM and turning up the "heat".

Wouldn't your own LLM be overkill? Ideally one would generate decoy junk more much efficiently than these abusive/hostile attackers can steal it.

Re: AI companies cause most of traffic on forums

#284

Post by alphan0n »

loeg wrote: post #194
mentalgear wrote: post #66

Note-worthy from the article (as some commentators suggested blocking them). "If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet."

I'd kind of like to see that claim substantiated a little more. Is it all crawlers that switch to a non-bot UA, or how are they determining it's the same bot? What non-bot UA do they claim?

I would take anything the author said with a grain of salt. They straight up lied about the configuration of the robots.txt file.

https://news.ycombinator.com/item?id=42551628

Re: AI companies cause most of traffic on forums

#285

Post by alphan0n »

andrethegiant wrote: post #254

CommonCrawl is supposed to help for this, i.e. crawl once and host the dataset for any interested party to download out of band. However, data can be up to a month stale, and it costs $$ to move the data out of us-east-1. I’m working on a centralized crawling platform[1] that aims to reduce OP’s problem. A caching layer with ~24h TTL for unauthed content would shield websites from redundant bot traffic while still pr…

Laughably, CommonCrawl shows that the authors robots.txt were configured to allow all, the entire time.

https://pastebin.com/VSHMTThJ

Re: AI companies cause most of traffic on forums

#286

Post by kmoser »

bodantogat wrote: post #89
markerz wrote: post #23

One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…

I see a lot of traffic I can tell are bots based on the URL patterns they access. They do not include the "bot" user agent, and often use residential IP pools. I haven't found an easy way to block them. They nearly took out my site a few days ago too.

My cheap and dirty way of dealing with bots like that is to block any IP address that accesses any URLs in robots.txt. It's not a perfect strategy but it gives me pretty good results given the simplicity to implement.

Re: AI companies cause most of traffic on forums

#287

Post by denschub »

alphan0n wrote: post #274
denschub wrote: post #258

Earlier quoted context omitted.

notice how there's a period of almost two months with no new index, just until a week before I posted this? I wonder what might have caused this!!1 (and it's not like they only check robots.txt once a month or so. https://stuff.overengineer.dev/stash/2024-12-30-dfwiki-opena... )

:/ Common Crawl archives robots.txt and indicates that the file at wiki.diasporafoundation.org was unchanged in November and December from what it is now. Unchanged from September, in fact. https://pastebin.com/VSHMTThJ https://index.commoncrawl.org/

just for you, I redeployed the old robots.txt (with an additional log-honeypot). I even manually submitted it to the web archive just now so you have something to look at: https://web.archive.org/web/20241231041718/https://wiki.dias...

they ingested it twice since I deployed it. they still crawl those URLs - and I'm sure they'll continue to do so - as others in that thread have confirmed exactly the same. I'll be traveling for the next couple of days, but I'll check the logs again when I'm back.

of course, I'll still see accessed from them, as most others in this thread do, too, even if they block them via robots.txt. but of course, that won't stop you from continuing to claim that "I lied". which, fine. you do you. luckily for me, there are enough responses from other people running medium-sized web stuffs with exactly the same observations, so I don't really care.

Re: AI companies cause most of traffic on forums

#288

Post by [deleted] »

alphan0n wrote: post #274
denschub wrote: post #258

Earlier quoted context omitted.

notice how there's a period of almost two months with no new index, just until a week before I posted this? I wonder what might have caused this!!1 (and it's not like they only check robots.txt once a month or so. https://stuff.overengineer.dev/stash/2024-12-30-dfwiki-opena... )

:/ Common Crawl archives robots.txt and indicates that the file at wiki.diasporafoundation.org was unchanged in November and December from what it is now. Unchanged from September, in fact. https://pastebin.com/VSHMTThJ https://index.commoncrawl.org/

[deleted]

Re: AI companies cause most of traffic on forums

#289

Post by alphan0n »

denschub wrote: post #287
alphan0n wrote: post #274

Earlier quoted context omitted.

:/ Common Crawl archives robots.txt and indicates that the file at wiki.diasporafoundation.org was unchanged in November and December from what it is now. Unchanged from September, in fact. https://pastebin.com/VSHMTThJ https://index.commoncrawl.org/

just for you, I redeployed the old robots.txt (with an additional log-honeypot). I even manually submitted it to the web archive just now so you have something to look at: https://web.archive.org/web/20241231041718/https://wiki.dias... they ingested it twice since I deployed it. they still crawl those URLs - and I'm sure they'll continue to do so - as others in that thread have confirmed exactly the same. I'll be tra…

What about the CommonCrawl archives? That clearly show the same robots.txt that allows all, from September through December?

You’re a phony.

Re: AI companies cause most of traffic on forums

#290

Post by denschub »

alphan0n wrote: post #289
denschub wrote: post #287

Earlier quoted context omitted.

just for you, I redeployed the old robots.txt (with an additional log-honeypot). I even manually submitted it to the web archive just now so you have something to look at: https://web.archive.org/web/20241231041718/https://wiki.dias... they ingested it twice since I deployed it. they still crawl those URLs - and I'm sure they'll continue to do so - as others in that thread have confirmed exactly the same. I'll be tra…

What about the CommonCrawl archives? That clearly show the same robots.txt that allows all, from September through December? You’re a phony.

Here's something for the next time you want to "expose" a phony: before linking me to your investigative source, ask for exact date-stamps when I made changes to the robots.txt and what I did, as well as when I blocked IPs. I could have told you those exactly, because all those changes are tracked in a git repo. If you asked me first, I could have answered you with the precise dates, and you would have realized that your whole theory makes absolutely no sense. Of course, that entire approach is mood now, because I'm not an idiot and I know when commoncrawl crawls, so I could easily adjust my response to their crawling dates, and you would of course claim I did.

So I'll just wear my "certified-phony-by-orangesite-user" badge with pride.

Take care, anonymous internet user.

Post reply on HN