AI companies cause most of traffic on forums • Hacker News Board

Re: AI companies cause most of traffic on forums

#41

Post by joshdavham » Mon, Dec 30, 2024, 5:27 PM UTC

I deployed a small dockerized app on GCP a couple months ago and these bots ended up costing me a ton of money for the stupidest reason: https://github.com/streamlit/streamlit/issues/9673

I originally shared my app on Reddit and I believe that that’s what caused the crazy amount of bot traffic.

Re: AI companies cause most of traffic on forums

#42

Post by ben_w » Mon, Dec 30, 2024, 5:27 PM UTC

walterbell wrote: post #20

brookst wrote: post #16

Earlier quoted context omitted.

The latter is clever but unlikely to do any harm. These companies spend a fortune on pre-training efforts and doubtlessly have filters to remove garbage text. There are enough SEO spam pages that just list nonsense words that they would have to.

Obfuscators can evolve alongside other LLM arms races.

Yes, but with an attacker having advantage because it directly improves their own product even in the absence of this specific motivation for obfuscation: any Completely Automated Public Turing test to tell Computers and Humans Apart can be used to improve the output of an AI by requiring the AI to pass that test.

And indeed, this has been part of the training process for at least some of OpenAI models before most people had heard of them.

Re: AI companies cause most of traffic on forums

#43

Post by bvan » Mon, Dec 30, 2024, 5:28 PM UTC

Need redirection to AI honeypots. Lore Ipsum ad infinitum.

Re: AI companies cause most of traffic on forums

#44

Post by mtu9001 » Mon, Dec 30, 2024, 5:28 PM UTC

kerblang wrote: post #13

It looks like various companies with resources are using available means to block AI bots - it's just that the little guys don't have that kinda stuff at their disposal. What does everybody use to avoid DDOS in general? Is it just becoming Cloudflare-or-else?

Cloudflare, Radware, Netscout, Cloud providers, perimeter devices, carrier null-routes, etc.

Re: AI companies cause most of traffic on forums

#45

Post by latenightcoding » Mon, Dec 30, 2024, 5:29 PM UTC

some of these companies are straight up inept. Not an AI company but "babbar.tech" was DDOSing my site, I blocked them and they still re-visit thousands of pages every other day even if it just returns a 404 for them.

Re: AI companies cause most of traffic on forums

#46

Post by GaggiX » Mon, Dec 30, 2024, 5:29 PM UTC

walterbell wrote: post #29

GaggiX wrote: post #26

Earlier quoted context omitted.

I imagine these companies today are curing their data with LLMs, this stuff isn't going to do anything.

Attackers don't have a monopoly on LLM expertise, defenders can also use LLMs for obfuscation. Technology arms races are well understood.

I hate LLM companies, I guess I'm going to use OpenAI API to "obfuscate" the content or maybe I will buy an NVIDIA GPU to run a llama model, mhm maybe on GPU cloud.

Re: AI companies cause most of traffic on forums

#47

Post by botanical76 » Mon, Dec 30, 2024, 5:30 PM UTC

GaggiX wrote: post #26

walterbell wrote: post #10

OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scraper... For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/ > The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the c…

I imagine these companies today are curing their data with LLMs, this stuff isn't going to do anything.

You're right, this approach is too easy to spot. Instead, pass all your blog posts through an LLM to automatically inject grammatically sound inaccuracies.

Re: AI companies cause most of traffic on forums

#48

Post by CoastalCoder » Mon, Dec 30, 2024, 5:30 PM UTC

markerz wrote: post #23

One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…

I wonder if it would work to send Meta's legal department a notice that they are not permitted to access your website.

Would that make subsequent accesses be violations of the U.S.'s Computer Fraud and Abuse Act?

Re: AI companies cause most of traffic on forums

#49

Post by coldpie » Mon, Dec 30, 2024, 5:30 PM UTC

markerz wrote: post #23

One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…

Imagine being one of the monsters who works at Facebook and thinking you're not one of the evil ones.

Re: AI companies cause most of traffic on forums

#50

Post by lowbloodsugar » Mon, Dec 30, 2024, 5:30 PM UTC

Neil44 wrote: post #12

johng wrote: post #2

If they ignore robots.txt there should be some kind of recourse :(

Error 403 is your only recourse.

Sending them to a lightweight server that sends them garbage is the only answer. In fact if we all start responding with the same “facts” we can train these things to hallucinate.