Of course Googlebot, Bingbot, Applebot, Amazonbot, YandexBot, etc from the major corps are HTTP useragent spiders that will have their downloaded public content used by corporations for AI training too. Might as well just drop the "AI" and say "corporate scrapers".
Miasma: A tool to trap AI web scrapers in an endless poison pit
71–80 of 276 posts
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#72I dunno... it feels like the same approach as those people who tell you gleeful stories of how they kept a phone spammer on a call for 45 minutes: "That'll teach 'em, ha ha!" Do these types of techniques really work? I’m not convinced.
Also, inserting hidden or misleading links is specifically a no-no for Google Search [0], who have this to say: We detect policy-violating practices both through automated systems and, as needed, human review that can result in a manual action. Sites that violate our policies may rank lower in results or not appear in results at all.
So you may well end up doing more damage to your own site than to the bots by using dodgy links in this manner.
[0]https://developers.google.com/search/docs/essentials/spam-po...
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#73> If you have a public website, they are already stealing your work. I have a public website, and web scrapers are stealing my work. I just stole this article, and you are stealing my comment. Thieves, thieves, and nothing but thieves!
I agree theft isn't a good analogy, but there is something similar going on. I put my words out into the world as a form of sharing. I enjoy reading things others write and share freely, so I write so others might enjoy the things I write. But now the things I write and share freely are being used to put money in the bank accounts of the worst people on the planet. They are using my work in a way I don't want it to be used. It makes me not want to share anymore.
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#74Earlier quoted context omitted.
Odd thing about cookies… they disappear after one serving. Websites are an endless stream of cookies. The analogy doesn’t hold.
Digital information may be our first post-scarce resource. It's interesting, and sad, to see so many attempt to fit it within scarcity-based economic models.
> digital information may be our first post-scarce resource
… browses memory and storage prices on NewEgg …
Hmm.
But the word digital is distracting us.
The word information is the important one. The question isn't where information goes. It's where information comes from.
Is new information post scarcity?
Can it ever be?
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#75> If you have a public website, they are already stealing your work. I have a public website, and web scrapers are stealing my work. I just stole this article, and you are stealing my comment. Thieves, thieves, and nothing but thieves!
I agree theft isn't a good analogy, but there is something similar going on. I put my words out into the world as a form of sharing. I enjoy reading things others write and share freely, so I write so others might enjoy the things I write. But now the things I write and share freely are being used to put money in the bank accounts of the worst people on the planet. They are using my work in a way I don't want it to b…
> But now the things I write and share freely are being used to put money in the bank accounts of the worst people on the planet.
I don't think that's the case. I'm not even arguing they aren't the worst people on the planet - might as well be. But all is see them doing is burning money all over the place.
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#76The data poisoning angle is interesting. Models trained on scraped web data inherit whatever biases, errors, and manipulation exist in that data. If bad actors can inject corrupted data at scale, it creates a malign incentive structure where model training becomes adversarial. The real solution is probably better data provenance -- models trained on licensed, curated datasets will eventually outcompete those trained on the open web.
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#77I did a related approach:
A toll charging gateway for llm scrapers: a modification to robots.txt to add price sheets in the comment field like a menu.
This was for a hackathon by forking certbot. Cloudflare has an enterprise version of this but this one would be self hosted
I think it has legs but I think I need to get pushed and goaded otherwise I tend to lose interest ...
It was for the USDC company btw so that's why there's a crypto angle - this might be a valid use case!
I'm open to crypto not all being hustles and scams
Tell me what you think?
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#78Earlier quoted context omitted.
If someone hands out cookies in the supermarket, are you allowed to grab everything and leave?
This is a dishonest analogy. In your example, there is only a limited amount of cookies available. While there is no practical limit on the amount of time a certain digital media can be viewed. You are allowed to take one cookie. But you are allowed to view a public website multiple times if you so want.
[flagged]
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#79> If you have a public website, they are already stealing your work. I have a public website, and web scrapers are stealing my work. I just stole this article, and you are stealing my comment. Thieves, thieves, and nothing but thieves!
The problem I have, is they hammer my site so hard they take it down.
The content is for everyone. They can have it. Just don't also take it away from everybody else.
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#80Earlier quoted context omitted.
I agree theft isn't a good analogy, but there is something similar going on. I put my words out into the world as a form of sharing. I enjoy reading things others write and share freely, so I write so others might enjoy the things I write. But now the things I write and share freely are being used to put money in the bank accounts of the worst people on the planet. They are using my work in a way I don't want it to b…
> But now the things I write and share freely are being used to put money in the bank accounts of the worst people on the planet. I don't think that's the case. I'm not even arguing they aren't the worst people on the planet - might as well be. But all is see them doing is burning money all over the place.
They’re getting the money to burn, though