Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

Post reply on HN
Print view View original

101–110 of 466 posts

Re: AI companies cause most of traffic on forums

#101

Post by markerz »

devit wrote: post #63
markerz wrote: post #23

One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…

[flagged]

Can't every webserver crash due to being overloaded? There's an upper limit to performance of everything. My website is a hobby and has a budget of $4/mo budget VPS.

Perhaps I'm saying crash and you're interpreting that as a bug but really it's just an OOM issue cause of too many in-flight requests. IDK, I don't care enough to handle serving my website at Facebook's scale.

Re: AI companies cause most of traffic on forums

#102

Post by iinnPP »

acedTrex wrote: post #76
azemetre wrote: post #70

Earlier quoted context omitted.

There’s a fine line between scrapping and DDOS’ing I’m sure. Just because you manufacture chemicals doesn’t mean you can legally dump your toxic waste anywhere you want (well shouldn’t be allowed to at least). You also shouldn’t be able to set your crawlers causing sites to fail.

intent is likely very important to something like a ddos charge

Wilful ignorance is generally enough.

Re: AI companies cause most of traffic on forums

#103

Post by uludag »

I'm always curious how poisoning attacks could work. Like, suppose that you were able to get enough human users to produce poisoned content. This poisoned content would be human written and not just garbage, and would contain flawed reasoning, misjudgments, lapses of reasoning, unrealistic premises, etc.

Like, I've asked ChatGPT certain questions where I know the online sources are limited and it would seem that from a few datapoints it can come up with a coherent answer. Imagine attacks where people would publish code misusing libraries. With certain libraries you could easily outnumber real data with poisoned data.

Re: AI companies cause most of traffic on forums

#104

Post by pogue »

walterbell wrote: post #72
GaggiX wrote: post #46

Earlier quoted context omitted.

I hate LLM companies, I guess I'm going to use OpenAI API to "obfuscate" the content or maybe I will buy an NVIDIA GPU to run a llama model, mhm maybe on GPU cloud.

With tiny amounts of forum text, obfuscation can be done locally with open models and local inference hardware (NPU on Arm SoC). Zero dollars sent to OpenAI, NVIDIA, AMD or GPU clouds.

What specifically are you suggesting? Is this a project that already exists or a theory of yours?

Re: AI companies cause most of traffic on forums

#105

Post by throwaway290 »

newsclues wrote: post #96
bodantogat wrote: post #89

Earlier quoted context omitted.

I see a lot of traffic I can tell are bots based on the URL patterns they access. They do not include the "bot" user agent, and often use residential IP pools. I haven't found an easy way to block them. They nearly took out my site a few days ago too.

The amateurs at home are going to give the big companies what they want: an excuse for government regulation.

If it doesn't say it's a bot and it doesn't come from a corporate IP it doesn't mean it's NOT a bot and not run by some "AI" company.

Re: AI companies cause most of traffic on forums

#106

Post by optimalsolver »

mentalgear wrote: post #66

Note-worthy from the article (as some commentators suggested blocking them). "If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet."

Ban evasion for me, but not for thee.

Re: AI companies cause most of traffic on forums

#107

Post by Mistletoe »

buro9 wrote: post #56

Their appetite cannot be quenched, and there is little to no value in giving them access to the content. I have data... 7d from a single platform with about 30 forums on this instance. 4.8M hits from Claude 390k from Amazon 261k from Data For SEO 148k from Chat GPT That Claude one! Wowser. Bots that match this (which is also the list I block on some other forums that are fully private by default): (?i). (AhrefsBot|AI…

This is a new twist on the Dead Internet Theory I hadn’t thought of.

Re: AI companies cause most of traffic on forums

#108

Post by nabla9 »

jandrese wrote: post #28
markerz wrote: post #23

One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…

If a bot ignores robots.txt that's a paddlin'. Right to the blacklist.

The linked article explains what happens when you block their IP.

Re: AI companies cause most of traffic on forums

#109

Post by 55555 »

arrowsmith wrote: post #69
jonnycomputer wrote: post #32

Earlier quoted context omitted.

how do you determine that they know the content of the honeypot?

Presumably the "honeypot" is an obscured link that humans won't click (e.g. tiny white text on a white background in a forgotten corner of the page) but scrapers will. Then you can determine whether a given IP visited the link.

I interpreted it to mean that a hidden page (linked as u describe) is indexed in Bing or that some "facts" written on a hidden page are regurgitated by ChatGPT.

Re: AI companies cause most of traffic on forums

#110

Post by jonnycomputer »

arrowsmith wrote: post #69
jonnycomputer wrote: post #32

Earlier quoted context omitted.

how do you determine that they know the content of the honeypot?

Presumably the "honeypot" is an obscured link that humans won't click (e.g. tiny white text on a white background in a forgotten corner of the page) but scrapers will. Then you can determine whether a given IP visited the link.

I know what a honeypot is, but the question is how the know the scraped data was actually used to train llms. I wondered whether they discovered or verified that by getting the llm to regurgitate content from the honeypot.
Post reply on HN