Funny thing is half these websites are probably served over cloud so Google, Amazon, and MSFT DDoS themselves and charge the clients for traffic.
AI companies cause most of traffic on forums
201–210 of 466 posts
Re: AI companies cause most of traffic on forums
#202Note-worthy from the article (as some commentators suggested blocking them). "If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet."
I'd kind of like to see that claim substantiated a little more. Is it all crawlers that switch to a non-bot UA, or how are they determining it's the same bot? What non-bot UA do they claim?
> Is it all crawlers that switch to a non-bot UA
I've observed only one of them do this with high confidence.
> how are they determining it's the same bot?
it's fairly easy to determine that it's the same bot, because as soon as I blocked the "official" one, a bunch of AWS IPs started crawling the same URL patterns - in this case, mediawiki's diff view (`/wiki/index.php?title=[page]&diff=[new-id]&oldid=[old-id]`), that absolutely no bot ever crawled before.
> What non-bot UA do they claim?
Latest Chrome on Windows.
Re: AI companies cause most of traffic on forums
#203I have a large forum with millions of posts that is frequently crawled and LLMs know a lot about it. It’s surprising how ChatGPT and company know about the history of the forum and pretty cool. But I also feel like it’s a fun opportunity to be a little mischievous and try to add some text to old pages that can sway LLMs somehow. Like a unique word. Any ideas?
It might be very interesting to check your current traffic against recent api outages at OpenAI. I have always wondered how many bots we have out there in the wild acting like real humans online. If usage dips during these times, it might be enlightening. https://x.com/mbrowning/status/1872448705124864178
I would expect AI APIs and AI scraping bots to run on separate infrastructures, so the latter wouldn’t necessarily be affected by outages of the former.
Re: AI companies cause most of traffic on forums
#204Note-worthy from the article (as some commentators suggested blocking them). "If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet."
I'd kind of like to see that claim substantiated a little more. Is it all crawlers that switch to a non-bot UA, or how are they determining it's the same bot? What non-bot UA do they claim?
Presumably they switch UA to Mozilla/something but tell on themselves by still using the same IP range or ASN. Unfortunately this has become common practice for feed readers as well.
Re: AI companies cause most of traffic on forums
#205I would be interested in people's thoughts here on my solution: https://www.tela.app.
The answer to bot spam: payments, per message.
I will soon be releasing a public forum system based on this model. You have to pay to submit posts.
Re: AI companies cause most of traffic on forums
#206Earlier quoted context omitted.
Crashing wasn't the intent. And scraping is legal, as I remember per Linkedin case.
If I make a physical robot and it runs someone over, I'm still liable, even though it was a delivery robot, not a running over people robot. If a bot sends so many requests that a site completely collapses, the owner is liable, even though it was a scraping bot and not a denial of service bot.
The law doesn't work by analogy.
Re: AI companies cause most of traffic on forums
#207Earlier quoted context omitted.
You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy. Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills. Maybe you don't even need a full LLM. Just a simple transfor…
Self plug, but I made this to deal with bots on my site: https://marcusb.org/hacks/quixotic.html . It is a simple markov generator to obfuscate content (static-site friendly, no server-side dynamic generation required) and an optional link-maze to send incorrigible bots to 100% markov-generated non-sense (requires a server-side component.)
I tested it on your site and I'm curious, is there a reason why the link-maze links are all gibberish (as in "oNvUcPo8dqUyHbr")? I would have had links be randomly inserted in the generated text going to "[random-text].html" so they look a bit more "real".
Re: AI companies cause most of traffic on forums
#208Can someone point out the authors robots.txt where the offense is taking place? I’m just seeing: https://pod.geraspora.de/robots.txt Which allows all user agents. *The discourse server does not disallow the offending bots mentioned in their post: https://discourse.diasporafoundation.org/robots.txt Nor does the wiki: https://wiki.diasporafoundation.org/robots.txt No robots.txt at all on the homepage: https://diasporaf…
the robots.txt on the wiki is no longer what it was when the bot accessed it. primarily because I clean up my stuff afterwards, and the history is now completely inaccessible to non-authenticated users, so there's no need to maintain my custom robots.txt.
Re: AI companies cause most of traffic on forums
#209Years ago I was building a search engine from scratch (back when that was a viable business plan). I was responsible for the crawler.
I built it using a distributed set of 10 machines with each being able to make ~1k queries per second. I generally would distribute domains as disparately as possible to decrease the load on machines.
Inevitably I'd end up crashing someone's site even though we respected robots.txt, rate limited, etc. I still remember the angry mail we'd get and how much we tried to respect it.
18 years later and so much has changed.
Re: AI companies cause most of traffic on forums
#210OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scraper... For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/ > The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the c…
I imagine these companies today are curing their data with LLMs, this stuff isn't going to do anything.
That opens up the opposite attack though: what do you need to do to get your content discarded by the AI?
I doubt you'd have much trouble passing LLM-generated text through their checks, and of course the requirements for you would be vastly different. You wouldn't need (near) real-time, on-demand work, or arbitrary input. You'd only need to (once) generate fake doppelganger content for each thing you publish.
If you wanted to, you could even write this fake content yourself if you don't mind the work. Feed Open AI all those rambling comments you had the clarity not to send.