AI companies cause most of traffic on forums • Hacker News Board

Re: AI companies cause most of traffic on forums

#261

Post by blibble » Mon, Dec 30, 2024, 8:38 PM UTC

rickyhatespeas wrote: post #68

brookst wrote: post #16

Earlier quoted context omitted.

The latter is clever but unlikely to do any harm. These companies spend a fortune on pre-training efforts and doubtlessly have filters to remove garbage text. There are enough SEO spam pages that just list nonsense words that they would have to.

It will do harm to their own site considering it's now un-indexable on platforms used by hundreds of millions and growing. Anyone using this is just guaranteeing that their content will be lost to history at worst, or just inaccessible to most search engines/users at best. Congrats on beating the robots, now every time someone searches for your site they will be taken straight to competitors.

I'd rather no-one read it and die forgotten than help "usher in the AI era"

Re: AI companies cause most of traffic on forums

#262

Post by throwaway_fai » Mon, Dec 30, 2024, 8:39 PM UTC

bongodongobob wrote: post #221

herval wrote: post #180

Earlier quoted context omitted.

A wordpress plugin that responds with lorem ipsum if the requester is a bot would also help poison the dataset beautifully

Nah, easily filtered out.

How about this, then. It's my (possibly incorrect) understanding that all the big LLM products still lose money per query. So you get a Web request from some bot, and on the backend, you query the corresponding LLM, asking it to generate dummy website content. Worm's mouth, meet worm's tail.

(I'm proposing this tongue in cheek, mostly, but it seems like it might work.)

Re: AI companies cause most of traffic on forums

#263

Post by okanat » Mon, Dec 30, 2024, 8:49 PM UTC

Aeolun wrote: post #199

yodsanklai wrote: post #19

Earlier quoted context omitted.

> court ruling Isn't this country dependent though?

Enforcement is not. What does the US care for what an EU court says about the legality of the OpenAI scraper.

They can charge the company continuously growing amounts in the EU and even ban a complete IP block if they don't fix their behavior.

Re: AI companies cause most of traffic on forums

#264

Post by gagik_co » Mon, Dec 30, 2024, 8:50 PM UTC

marcus0x62 wrote: post #161

echelon wrote: post #129

Earlier quoted context omitted.

You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy. Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills. Maybe you don't even need a full LLM. Just a simple transfor…

Self plug, but I made this to deal with bots on my site: https://marcusb.org/hacks/quixotic.html . It is a simple markov generator to obfuscate content (static-site friendly, no server-side dynamic generation required) and an optional link-maze to send incorrigible bots to 100% markov-generated non-sense (requires a server-side component.)

This is cool! It'd have been funny for this to become mainstream somehow and mess with LLM progression. I guess that's already happening with all the online AI slop that is being re-fed into its training.

Re: AI companies cause most of traffic on forums

#265

Post by joshdavham » Mon, Dec 30, 2024, 9:02 PM UTC

Aeolun wrote: post #166

watermelon0 wrote: post #155

Earlier quoted context omitted.

That would still be 86k req/day, which can be quite expensive in a serverless environment, especially if the app is not optimized.

That’s a problem of the serverless environment, not of not being a good netizen. Seriously, my toaster from 20 years ago could serve 1req/s

What would you recommend I do instead? Deploying a Docker container on Cloud Run sorta seemed like the logical way to deploy my micro app.

Also for more context, this was the app in question (now moved to streamlit cloud): https://jreadability-demo.streamlit.app/

Re: AI companies cause most of traffic on forums

#266

Post by endofreach » Mon, Dec 30, 2024, 9:04 PM UTC

echelon wrote: post #129

bodantogat wrote: post #89

Earlier quoted context omitted.

I see a lot of traffic I can tell are bots based on the URL patterns they access. They do not include the "bot" user agent, and often use residential IP pools. I haven't found an easy way to block them. They nearly took out my site a few days ago too.

You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy. Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills. Maybe you don't even need a full LLM. Just a simple transfor…

> Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills

Or just wait for after the AI flood has peaked & most easily scrapable content has been AI generated (or at least modified).

We should seriously start discussing the future of the public web & how to not leave it to big tech before it's too late. It's a small part of something i am working on, but not central. So i haven't spend enough time to have great answers. If anyone reading this seriously cares, i am waiting desperately to exchange thoughts & approaches on this.

Re: AI companies cause most of traffic on forums

#267

Post by x0x0 » Mon, Dec 30, 2024, 9:16 PM UTC

ndriscoll wrote: post #229

troupo wrote: post #216

Earlier quoted context omitted.

You completely ignore the fact that they are also requesting a lot of pages that can be expensive to retrieve/calculate.

Beyond something like running an ML model, what web pages are expensive (enough that 1-10 requests/second matters at all) to generate these days?

I've worked on multiple sites like this over my career.

Our pages were expensive to generate, so what scraping did is blew out all our caches by yanking cold pages/images into memory. Page caches, fragment caches, image caches, but also the db working set in ram, making every single thing on the site slow.

Re: AI companies cause most of traffic on forums

#268

Post by oblio » Mon, Dec 30, 2024, 9:25 PM UTC

nedrocks wrote: post #213

ai-christianson wrote: post #122

Earlier quoted context omitted.

Would you consider giving these crawlers access if they paid you?

This is one of the few interesting uses of crypto transactions at reasonable scale in the real world.

Use the ex-hype to kill the new hype?

And the ex-hype would probably fail at that, too :-)

Re: AI companies cause most of traffic on forums

#269

Post by oblio » Mon, Dec 30, 2024, 9:26 PM UTC

IanKerr wrote: post #164

mentalgear wrote: post #66

Note-worthy from the article (as some commentators suggested blocking them). "If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet."

This is the beginning of the end of the public internet, imo. Websites that aren't able to manage the bandwidth consumption of AI scrapers and the endless spam that will take over from LLMs writing comments on forums are going to go under. The only things left after AI has its way will be walled gardens with whitelisted entrants or communities on large websites like Facebook. Niche, public sites are going to become u…

Classic spam all but killed small email hosts, AI spam will kill off the web.

Super sad.

Re: AI companies cause most of traffic on forums

#270

Post by tivert » Mon, Dec 30, 2024, 9:27 PM UTC

echelon wrote: post #129

bodantogat wrote: post #89

Earlier quoted context omitted.

I see a lot of traffic I can tell are bots based on the URL patterns they access. They do not include the "bot" user agent, and often use residential IP pools. I haven't found an easy way to block them. They nearly took out my site a few days ago too.

You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy. Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills. Maybe you don't even need a full LLM. Just a simple transfor…

> You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy.

> Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills.

I agree, and not just to discourage them running up traffic bills. The end-state of what they hope to build is very likely to be extremely for most regular people [1], so we shouldn't cooperate in building it.

[1] And I mean end state. I don't care how much value you say you get from some AI coding assistant today, the end state is your employer happily gets to fire you and replace you with an evolved version of the assistant at a fraction of your salary. The goal is to eliminate the cost that is our livelihoods. And if we're lucky, in exchange we'll get a much reduced basic income sufficient to count the rest of our days from a dense housing project filled with cheap minimum-quality goods and a machine to talk to if we're sad.