Love it. Thanks for doing this work. Not sure why people are criticizing this. Also, insane amount of work has been done to improve scraping - which in my mind is just absolute bonkers and i didn't see people complaining about that.
Miasma: A tool to trap AI web scrapers in an endless poison pit
131–140 of 276 posts
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#132Earlier quoted context omitted.
Unintentional denial-of-service attacks from AI scrapers are definitely a problem, I just don't know if "theft" is the right way to classify them. They shouldn't get lumped in with intellectual property concerns, which are a different matter. AI scrapers are a tragedy of the commons problem kind of like Kessler syndrome: a few bad actors can ruin low Earth orbit for everyone via space pollution, which is definitely a…
you're totally right about not being theft, but we have a term. you used it yourself, "distributed denial of service". that's all it is. these crawlers should be kicked off the internet for abuse. people should contact the isp of origin.
Firstly, since this argument is about semantic pedantry anyways, it's just denial-of-service, not distributed denial-of-service. AI scraper requests come from centralized servers, not a botnet.
Secondly, denial-of-service implies intentionality and malice that I don't think is present from AI scrapers. They cause huge problems, but only as a negligent byproduct of other goals. I think that the tragedy of the commons framing is more accurate.
EDIT: my first point was arguably incorrect because some scrapers do use decentralized infrastructure and my second point was clearly incorrect because "denial-of-service" describes the effect, not the intention. I retract both points and apologize.
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#133Way back in the day I had a software product, with a basic system to prevent unauthorised sharing, since there was a small charge for it. Every time I released an update, and new crack would appear. For the next six months I worked on improving the anti-copying code until I stumbled across an article by a coder in the same boat as me. He realised he was now playing a game with some other coders where he make the copy…
> the cracker would then have fun cracking it. I wonder if you could've won by making the cracking boring. No new techniques, bare minimum changes to require compiling a new crack, and just enough to make it difficult to automate. I.e. turn the cracking into a job. But in reality, there are other community-driven motivations to put out cracks.
>No new techniques, bare minimum changes to require compiling a new crack, and just enough to make it difficult to automate.
From a practical perspective you also have to have a steady stream of features for the newer versions to be worth cracking. Otherwise why use v1.09 when v1.01 works fine? Moreover spending less effort into improving the DRM is still playing at the cat and mouse game, albeit with less time investment. If you're making minimal changes, the cracker also has to spend minimal time updating the crack.
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#134Earlier quoted context omitted.
> This is ultimately just going to give them training material for how to avoid this crap. > The arms race just took another step, and if you're spending money creating or hosting this kind of content, it's not going to make up for the money you're losing by your other content getting scraped. So we should all just do nothing and accept the inevitable?
> So we should all just do nothing and accept the inevitable? I daresay rate-limiting will result in better outcomes than well-poisoning with hidden links that are against the policies of search engines. Lots of potential for collateral damage, including your own websites' reputations and search visibility, with the well-poisoning approach.
To be honest who cares about Google search anymore it's pretty useless these days.
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#135A related technique used to work so well for search engine spiders. I had some software i wrote called 'search engine cloaker'... this was back in the early 2000s... one of the first if not the first to do the shadowy "cloaking" stuff! We'd spin dummy content from lists of keywords and it was just piles and piles. We made it a bit smarter using Markov chains to make the sentences somewhat sensible. We'd auto-interlin…
You didn't 'give the spiders what they seem to want.' You exploited a naive ranking algorithm to inject garbage into search results that real people were trying to use. That you rationalized it at the time is human. That you're still rationalizing it decades later is something else.
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#136I did a related approach: A toll charging gateway for llm scrapers: a modification to robots.txt to add price sheets in the comment field like a menu. This was for a hackathon by forking certbot. Cloudflare has an enterprise version of this but this one would be self hosted I think it has legs but I think I need to get pushed and goaded otherwise I tend to lose interest ... It was for the USDC company btw so that's w…
This is literally what HTTP 402 is for -- there's a whole buncha work going on ... but please, please, please don't let Cloudflare become another bloody gatekeeper. Please.
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#137I dunno... it feels like the same approach as those people who tell you gleeful stories of how they kept a phone spammer on a call for 45 minutes: "That'll teach 'em, ha ha!" Do these types of techniques really work? I’m not convinced. Also, inserting hidden or misleading links is specifically a no-no for Google Search [0], who have this to say: We detect policy-violating practices both through automated systems and,…
Who TF cares about google? This is mostly for personal tech stuff (just the stuff AI steals for training). Id say its pretty welcome that it is not shown in google results.
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#138> If you have a public website, they are already stealing your work. I have a public website, and web scrapers are stealing my work. I just stole this article, and you are stealing my comment. Thieves, thieves, and nothing but thieves!
The problem I have, is they hammer my site so hard they take it down. The content is for everyone. They can have it. Just don't also take it away from everybody else.
Been there recently. Rate limit on nginx and anti-syn flood on pf solved it.
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#139I dunno... it feels like the same approach as those people who tell you gleeful stories of how they kept a phone spammer on a call for 45 minutes: "That'll teach 'em, ha ha!" Do these types of techniques really work? I’m not convinced. Also, inserting hidden or misleading links is specifically a no-no for Google Search [0], who have this to say: We detect policy-violating practices both through automated systems and,…
>I dunno... it feels like the same approach as those people who tell you gleeful stories of how they kept a phone spammer on a call for 45 minutes: "That'll teach 'em, ha ha!" Do these types of techniques really work? I’m not convinced If you are automating it, I don't see why not. Kitboga, a you-tuber kept scam callers in AI call-center loops tying up there resources so they cant use them on unsuspecting victims.[0]…
Pretty easy. Get a paid number and have the phone scammers / marketers call that. I know a guy who made a decent side huzzle from this. They marketers slowly blocked his number tho, not sure if he still has this thing going on, as it was more a experiment.
Re: Miasma: A tool to trap AI web scrapers in an endless poison pit
#140Can't we simple parse and remove any style="display: none;", aria-hidden="true", and tabindex="1" attributes before the text is processed and get around this trick? What am I missing?
Just have the link visible, but css it so that its either small as hell, or just off screen. Google / bots will follow it, real peopple will never see it.