Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI companies cause most of traffic on forums

pod.geraspora.de

Post reply on HN
Print view View original

211–220 of 466 posts

Re: AI companies cause most of traffic on forums

#211

Post by sangnoir »

brookst wrote: post #86
exe34 wrote: post #30

Earlier quoted context omitted.

if you can detect them, maybe feed them low iq stuff from a small llama. add latency to waste their time.

It would cost you more than it costs them. And there is enough low IQ stuff from humans that they already do tons of data cleaning.

> And there is enough low IQ stuff from humans that they already do tons of data cleaning

Whatever cleaning they do is not effective, simply because it cannot scale with the sheer volumes if data they ingest. I had an LLM authoritatively give an incorrect answer, and when I followed up to the source, it was from a fanfic page.

Everyone ITT who's being told to give up because its hopeless to defend against AI scrapers - you're being propagandized, I won't speculate on why - but clearly this is an arms race with no clear winner yet. Defenders are free to use LLM to generate chaff.

Re: AI companies cause most of traffic on forums

#212

Post by marcus0x62 »

gs17 wrote: post #207
marcus0x62 wrote: post #161

Earlier quoted context omitted.

Self plug, but I made this to deal with bots on my site: https://marcusb.org/hacks/quixotic.html . It is a simple markov generator to obfuscate content (static-site friendly, no server-side dynamic generation required) and an optional link-maze to send incorrigible bots to 100% markov-generated non-sense (requires a server-side component.)

I tested it on your site and I'm curious, is there a reason why the link-maze links are all gibberish (as in "oNvUcPo8dqUyHbr")? I would have had links be randomly inserted in the generated text going to "[random-text].html" so they look a bit more "real".

Its unfinished. At the moment, the links are randomly generated because that was an easy way to get a bunch of unique links. Sooner or later, I’ll just get a few tokens from the markov generator and use those for the link names.

I’d also like to add image obfuscation on the static generator side - as it stands now, anything other than text or html gets passed through unchanged.

Re: AI companies cause most of traffic on forums

#213

Post by nedrocks »

ai-christianson wrote: post #122
buro9 wrote: post #56

Their appetite cannot be quenched, and there is little to no value in giving them access to the content. I have data... 7d from a single platform with about 30 forums on this instance. 4.8M hits from Claude 390k from Amazon 261k from Data For SEO 148k from Chat GPT That Claude one! Wowser. Bots that match this (which is also the list I block on some other forums that are fully private by default): (?i). (AhrefsBot|AI…

Would you consider giving these crawlers access if they paid you?

This is one of the few interesting uses of crypto transactions at reasonable scale in the real world.

Re: AI companies cause most of traffic on forums

#215

Post by coldpie »

qazxcvbnmlp wrote: post #200
buro9 wrote: post #94

Earlier quoted context omitted.

I have thought about writing such a thing... 1. A proxy that looks at HTTP Headers and TLS cipher choices 2. An allowlist that records which browsers send which headers and selects which ciphers 3. A dynamic loading of the allowlist into the proxy at some given interval New browser versions or updates to OSs would need the allowlist updating, but I'm not sure it's that inconvenient and could be done via GitHub so peo…

Everything on this can be programmatically simulated by a bot with bad intentions. It will be a cat and mouse game of finding behaviors that differentiate between bot and not and patching them. To truly say “I trust real browsers” requires a signal of integrity of the user and browser such as cryptographic device attestation of the browser. .. which has to be centrally verified. Which is also not great.

> Everything on this can be programmatically simulated by a bot with bad intentions. It will be a cat and mouse game of finding behaviors that differentiate between bot and not and patching them.

Forcing Facebook & Co to play the adversary role still seems like an improvement over the current situation. They're clearly operating illegitimately if they start spoofing real user agents to get around bot blocking capabilities.

Re: AI companies cause most of traffic on forums

#216

Post by troupo »

ndriscoll wrote: post #152
markerz wrote: post #101

Earlier quoted context omitted.

Can't every webserver crash due to being overloaded? There's an upper limit to performance of everything. My website is a hobby and has a budget of $4/mo budget VPS. Perhaps I'm saying crash and you're interpreting that as a bug but really it's just an OOM issue cause of too many in-flight requests. IDK, I don't care enough to handle serving my website at Facebook's scale.

I wouldn't expect it to crash in any case, but I'd generally expect that even an n100 minipc should bottleneck on the network long before you manage to saturate CPU/RAM (maybe if you had 10Gbit you could do it). The linked post indicates they're getting ~2 requests/second from bots, which might as well be zero. Even low powered modern hardware can do thousands to tens of thousands.

You completely ignore the fact that they are also requesting a lot of pages that can be expensive to retrieve/calculate.

Re: AI companies cause most of traffic on forums

#217

Post by ku1ik »

npiano wrote: post #205

I would be interested in people's thoughts here on my solution: https://www.tela.app . The answer to bot spam: payments, per message. I will soon be releasing a public forum system based on this model. You have to pay to submit posts.

This is interesting!

Re: AI companies cause most of traffic on forums

#218

Post by ActVen »

layer8 wrote: post #203
ActVen wrote: post #134

Earlier quoted context omitted.

It might be very interesting to check your current traffic against recent api outages at OpenAI. I have always wondered how many bots we have out there in the wild acting like real humans online. If usage dips during these times, it might be enlightening. https://x.com/mbrowning/status/1872448705124864178

I would expect AI APIs and AI scraping bots to run on separate infrastructures, so the latter wouldn’t necessarily be affected by outages of the former.

Definitely. I'm just talking about an interesting way to identify content creation on a site.

Re: AI companies cause most of traffic on forums

#219

Post by bongodongobob »

bloppe wrote: post #55

They're the ones serving the expensive traffic. Wut if people were to form a volunteer bot net to waste their GPU resources in a similar fashion, just sending tons of pointless queries per day like "write me a 1000 word essay that ...". Could even form a non-profit around it and call it research.

... how do you plan on doing this without paying?

Re: AI companies cause most of traffic on forums

#220

Post by npiano »

ku1ik wrote: post #217
npiano wrote: post #205

I would be interested in people's thoughts here on my solution: https://www.tela.app . The answer to bot spam: payments, per message. I will soon be releasing a public forum system based on this model. You have to pay to submit posts.

This is interesting!

Thanks! Honestly, I think this approach is inevitable given the rising tide of unstoppable AI spam.
Post reply on HN