Why not simply blacklist or rate limit those bot IP’s ?
There are way too many to do that
Miasma: A tool to trap AI web scrapers in an endless poison pit
21–30 of 276 posts
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#22Why do this though? It's like if someone was trying to "trap" search crawlers back in the early 2000s. Seems counterproductive
Because of bots that don't respect ROBOTS.txt .
If you want an AI bot to crawl your website while you pay for that bandwidth then you wont use the tool.
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#23Why not simply blacklist or rate limit those bot IP’s ?
Because punishment for breaking the robots.txt rules is a social good.
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#24Why do this though? It's like if someone was trying to "trap" search crawlers back in the early 2000s. Seems counterproductive
Web crawlers didn’t routinely take down public resources or use the scraped info to generate facsimiles that people are still having ethical debates over. Its presence didn’t even register and it was indexing that helped them. It isn’t remotely the same thing.
https://www.libraryjournal.com/story/ai-bots-swarm-library-c...
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#25[deleted]
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#26[dead]
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#27Why not simply blacklist or rate limit those bot IP’s ?
If you have real traffic and bot traffic, you still need to identify which is which. On top of that, bots very likely don’t reuse the same IPs over and over again. I assume if we knew all the IPs used only by bots ahead of time, then yeah it would be simple to blacklist them. But although it’s simple in theory, the practice of identifying what to blacklist in the first place is the part that isn’t as simple
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#28> If you have a public website, they are already stealing your work.
I have a public website, and web scrapers are stealing my work. I just stole this article, and you are stealing my comment. Thieves, thieves, and nothing but thieves!
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#29Is there any evidence or hints that these actually work? It seems pretty reasonable that any scraper would already have mitigations for things like this as a function of just being on the internet.
it won't work, especially on gemini. Googlebot is very experienced when it comes to crawling. It might work for OpenAI and others maybe.
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#30Earlier quoted context omitted.
What kind of migitations? How would you detect the poison fountain?
style="display: none;" aria-hidden="true" tabindex="1" many scraper already know not to follow these, as it's how site used to "cheat" pagerank serving keyword soups
Google will give your website a penalty for doing this.