What if people used a kind of reverse slow-loris attack? Meaning, AI bot connects, and your site dribbles out content very slowly, just fast enough to keep the bot from timing out and disconnecting. And of course the output should be garbage.
AI companies cause most of traffic on forums
61–70 of 466 posts
Re: AI companies cause most of traffic on forums
#62Naive question, do people no longer respect robots.txt?
Re: AI companies cause most of traffic on forums
#63One of my websites was absolutely destroyed by Meta's AI bot: Meta-ExternalAgent https://developers.facebook.com/docs/sharing/webmasters/web-... It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again. My solution was to add…
[flagged]
Re: AI companies cause most of traffic on forums
#64Seems like many of these "AI companies" wouldn't need another funding round if they would do scraping ... (ironically) more intelligently.
Really, this behaviour should be a big embarrassment for any company whose main business model is selling "intelligence" as an outside product.
Re: AI companies cause most of traffic on forums
#65Their appetite cannot be quenched, and there is little to no value in giving them access to the content. I have data... 7d from a single platform with about 30 forums on this instance. 4.8M hits from Claude 390k from Amazon 261k from Data For SEO 148k from Chat GPT That Claude one! Wowser. Bots that match this (which is also the list I block on some other forums that are fully private by default): (?i). (AhrefsBot|AI…
You know, at this point, I wonder if an allowlist would work better.
Re: AI companies cause most of traffic on forums
#66Note-worthy from the article (as some commentators suggested blocking them).
"If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet."
Re: AI companies cause most of traffic on forums
#67OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scraper... For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/ > The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the c…
> OpenAI publishes IP ranges for their bots... If blocking them becomes standard practice, how long do you think it'd be before they started employing third-party crawling contractors to get data sets?
Maybe they want sites to block them that don't want to be crawled since it probably saves them a lawsuit down the road.
Re: AI companies cause most of traffic on forums
#68OpenAI publishes IP ranges for their bots, https://github.com/greyhat-academy/lists.d/blob/main/scraper... For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/ > The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the c…
The latter is clever but unlikely to do any harm. These companies spend a fortune on pre-training efforts and doubtlessly have filters to remove garbage text. There are enough SEO spam pages that just list nonsense words that they would have to.
It will do harm to their own site considering it's now un-indexable on platforms used by hundreds of millions and growing. Anyone using this is just guaranteeing that their content will be lost to history at worst, or just inaccessible to most search engines/users at best. Congrats on beating the robots, now every time someone searches for your site they will be taken straight to competitors.
Re: AI companies cause most of traffic on forums
#69Earlier quoted context omitted.
I exclude all the published LLM User-Agents and have a content honeypot on my website. Google obeys, but ChatGPT and Bing still clearly know the content of the honeypot.
how do you determine that they know the content of the honeypot?
Presumably the "honeypot" is an obscured link that humans won't click (e.g. tiny white text on a white background in a forgotten corner of the page) but scrapers will. Then you can determine whether a given IP visited the link.
Re: AI companies cause most of traffic on forums
#70Earlier quoted context omitted.
I wonder if it would work to send Meta's legal department a notice that they are not permitted to access your website. Would that make subsequent accesses be violations of the U.S.'s Computer Fraud and Abuse Act?
Crashing wasn't the intent. And scraping is legal, as I remember per Linkedin case.
There’s a fine line between scrapping and DDOS’ing I’m sure.
Just because you manufacture chemicals doesn’t mean you can legally dump your toxic waste anywhere you want (well shouldn’t be allowed to at least).
You also shouldn’t be able to set your crawlers causing sites to fail.