[deleted]
We can't have nice things because of AI scrapers
141–150 of 279 posts
Re: We can't have nice things because of AI scrapers
#142Earlier quoted context omitted.
The mechanism is putting some text that points to the downloads.
So perhaps it's time to standardize that .
I'm in favor of /.well-known/[ai|llm].txt or even a JSON or (gasp!) XML.
Or even /.well-known/ai/$PLATFORM.ext which would have the instructions.
Could even be "bootstrapped" from /robots.txt
Re: We can't have nice things because of AI scrapers
#143Earlier quoted context omitted.
I use iocaine[0] to generate a tarpit. Yesterday it served ~278k "pages" consisting of ~500MB of gibberish (and that's despite banning most AI scrapers in robots.txt.) [0] https://iocaine.madhouse-project.org
Can't seem to access this. It flashes some text briefly then gives me an 418 TEAPOT response. I wonder if it's because I'm on Linux? EDIT: Begrudgingly checked Chrome, and it loads. I guess it doesn't like Firefox?
Nor Safari on iOS.
Re: We can't have nice things because of AI scrapers
#144Earlier quoted context omitted.
I actually didnt know it was using a local model and that it fetches it locally.
They should advertise that. I pretty much reflexively avoid any mention of AI in interfaces because they usually mean "we're sending this all to openthropigoogosoft so I hope you don't have any secrets."
> They should advertise that
Previews can optionally include AI-generated key points, which are processed on your device to protect your privacy.
I'll also add that if you go to the Labs page (in settings) you can enable another local model to semantically search your history
Re: We can't have nice things because of AI scrapers
#145Cloudflare has a service for this now that will detect AI scrapers and send them to a tarpit of infinite AI generated nonsense pages.
Not sure "TLS added and removed here :)" as a Service is the right tool in the drawer for this.
Re: We can't have nice things because of AI scrapers
#146The SQLite team faced a similar problem last year, and Richard Hipp (the creator of SQLite) made almost the same comment: "The malefactor behind this attack could just clone the whole SQLite source repository and search all the content on his own machine, at his leisure. But no: Being evil, the culprit feels compelled to ruin it for everyone else. This is why you don't get to keep nice things...." https://sqlite.org/…
[flagged]
"The malefactor behind this attack" isn't a complaint about the web crawler.
Re: We can't have nice things because of AI scrapers
#147Earlier quoted context omitted.
> They won't believe a random site when it says "Look, stop hitting our API, you can pick all of this data in one go, over in this gzipped tar file." What mechanism does a site have for doing that? I don't see anything in robots.txt standard about being able to set priority but I could be missing something.
The only real mechanism is "Disallow: /rendered/pages/*" and "Allow: /archive/today.gz" or whatever and there is no communication that the latter is the former. There is no machine-standard AFAIK that allows webmasters to communicate to bot operators in this detail. It would be pretty cool if standard CMSes had such a protocol to adhere to. Install a plugin and people could 'crawl' your Wordpress from a single dump o…
A sitemap.xml file could get you most of the way there.
Re: We can't have nice things because of AI scrapers
#148How do they get overloaded? Is the website too slow? I have a quite big wiki online and barely see any impact from bots.
In these discussions no one will admit this, but the answer is generally yes. Websites written in python and stuff like that.
It's not "written too slow" if you e.g. only get 50 users a week, though. If bots add so much load that you need to go optimise your website for them, then that's a bot problem not a website problem.
Re: We can't have nice things because of AI scrapers
#149the more time passes the more i'm convinced that the solution is to—somehow—force everyone to have to go through something like common crawl i don't want people's servers to be pegged at 100% because a stupid dfs scraper is exhaustively traversing their search facets, but i also want the web to remain scrapable by ordinary people, or rather go back to how readily scrapable it used to be before the invention of cloudf…
That's sorta what MetaBrainz did - they offer their whole DB as a single tarball dump, much like what Wikipedia does. I downloaded it in the order of an hour; if I need a MusicBrainz lookup, I just do a local query. For this strategy to work, people need to actually use the DB dumps instead of just defaulting to scraping. Unfortunately scraping is trivially easy, particularly now that AI code assistants can write a w…
I mean this AI data scrapper would need to scan and fetch billions of website
why would they even care over 1 single website ??? You expect instiution to care out of billions website they must scrape daily
Re: We can't have nice things because of AI scrapers
#150Metabrainz is a great resource -- I wrote about them a few years ago here: https://www.eff.org/deeplinks/2021/06/organizing-public-inte... There's something important here in that a public good like Metabrainz would be fine with the AI bots picking up their content -- they're just doing it in a frustratingly inefficient way. It's a co-ordination problem: Metabrainz assumes good intent from bots, and has to lock down…
> The bots have a different model -- they assume that the website is adversarially "hiding" its content.
this should give us pause. if a bot considers this adversarial and is refusing to respect the site owners wishes, thats a big part of the problem.
a bot should not consider that “adversarial”