AI continues to ruin the entire internet.
AI companies cause most of traffic on forums
191–200 of 466 posts
Re: AI companies cause most of traffic on forums
#192I'm always curious how poisoning attacks could work. Like, suppose that you were able to get enough human users to produce poisoned content. This poisoned content would be human written and not just garbage, and would contain flawed reasoning, misjudgments, lapses of reasoning, unrealistic premises, etc. Like, I've asked ChatGPT certain questions where I know the online sources are limited and it would seem that from…
Unless a substantial portion of the internet starts serving poisoned content to bots, that won’t solve the bandwidth problem. And even if a substantial portion of the internet would start poisoning, bots would likely just shift to disguising themselves so they can’t be identified as bots anymore. Which according to the article they already do now when they are being blocked.
Re: AI companies cause most of traffic on forums
#193Their appetite cannot be quenched, and there is little to no value in giving them access to the content. I have data... 7d from a single platform with about 30 forums on this instance. 4.8M hits from Claude 390k from Amazon 261k from Data For SEO 148k from Chat GPT That Claude one! Wowser. Bots that match this (which is also the list I block on some other forums that are fully private by default): (?i). (AhrefsBot|AI…
Would you consider giving these crawlers access if they paid you?
Interesting idea, though I doubt they'd ever offer a reasonable amount for it. But doesn't it also change a sites legal stance if you're now selling your users content/data? I think it would also repel a number of users away from your service
Re: AI companies cause most of traffic on forums
#194Note-worthy from the article (as some commentators suggested blocking them). "If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet."
I'd kind of like to see that claim substantiated a little more. Is it all crawlers that switch to a non-bot UA, or how are they determining it's the same bot? What non-bot UA do they claim?
Re: AI companies cause most of traffic on forums
#195I'm always curious how poisoning attacks could work. Like, suppose that you were able to get enough human users to produce poisoned content. This poisoned content would be human written and not just garbage, and would contain flawed reasoning, misjudgments, lapses of reasoning, unrealistic premises, etc. Like, I've asked ChatGPT certain questions where I know the online sources are limited and it would seem that from…
(I was going to post "run a bot motel" as a topline, but I get tired of sounding like broken record.)
To generate garbage data I've had good success using Markov Chains in the past. These days I think I'd try an LLM and turning up the "heat".
Re: AI companies cause most of traffic on forums
#196I have a hypothetical question: lets say I want to slightly scramble the content of my site (no so much so as to be obvious, but enough that most knowledge within is lost) when I detect that a request is coming from one of these bots, could I face legal repercussions?
I can see two cases where it could be legally questionable:
- the result breaks some law (e.g. support of selected few genocidal regimes)
- you pretend users (people, companies) wrote something they didn't
Re: AI companies cause most of traffic on forums
#197Earlier quoted context omitted.
The latter is clever but unlikely to do any harm. These companies spend a fortune on pre-training efforts and doubtlessly have filters to remove garbage text. There are enough SEO spam pages that just list nonsense words that they would have to.
It will do harm to their own site considering it's now un-indexable on platforms used by hundreds of millions and growing. Anyone using this is just guaranteeing that their content will be lost to history at worst, or just inaccessible to most search engines/users at best. Congrats on beating the robots, now every time someone searches for your site they will be taken straight to competitors.
You can still fine-tune though. I often run User-Agent: *, Disallow: / with User-Agent: Googlebot, Allow: / because I just don't care for Yandex or baidu to crawl me for the 1 user/year they'll send (of course this depends on the region you're offering things to).
That other thing is only a more extreme form of the same thing for those who don't behave. And when there's a clear value proposition in letting OpenAI ingest your content you can just allow them to.
Re: AI companies cause most of traffic on forums
#198Earlier quoted context omitted.
Their problem is they can’t detect which are bots in the first place. If they could, they’d block them.
Then have the users solve ARC-AGI or whatever nonsense. If the bots want your content, they'll have to solve $3,000 of compute to get it.
That only works until The benchmark questions and answers are public. Which they necessarily would be in this case.
Re: AI companies cause most of traffic on forums
#199Earlier quoted context omitted.
court ruling a few years ago said it's legal to scrape web pages, you don't need to be respectful of these for any purely legal reasons however this doesn't stop the website from doing what they can to stop scraping attempts, or using a service to do that for them
> court ruling Isn't this country dependent though?
Enforcement is not. What does the US care for what an EU court says about the legality of the OpenAI scraper.
Re: AI companies cause most of traffic on forums
#200Earlier quoted context omitted.
You know, at this point, I wonder if an allowlist would work better.
I have thought about writing such a thing... 1. A proxy that looks at HTTP Headers and TLS cipher choices 2. An allowlist that records which browsers send which headers and selects which ciphers 3. A dynamic loading of the allowlist into the proxy at some given interval New browser versions or updates to OSs would need the allowlist updating, but I'm not sure it's that inconvenient and could be done via GitHub so peo…
Everything on this can be programmatically simulated by a bot with bad intentions. It will be a cat and mouse game of finding behaviors that differentiate between bot and not and patching them.
To truly say “I trust real browsers” requires a signal of integrity of the user and browser such as cryptographic device attestation of the browser. .. which has to be centrally verified. Which is also not great.