151–160 of 279 posts
Earlier quoted context omitted.
Unfortunately, Cloudflare often destroys the experience for users with shared connections, VPNs, exotic browsers… I had to remove it from my site after too many complaints.
I am sure Cloudflare would have no problem selling you a VPN service. After all, it's not very far from hosting booters and selling DoS protection.
Price is $5/mo
Nowadays people complain about AI scrapers with the same vain as they complained about search indexers a way back when. Just a few years later, people had stopped caring too much about storage access and bandwidth, and started begging search engines to visit their websites. Every trick on the planet Earth, SEO optimization, etc. Looking forward to the time when everybody suddenly starts to embrace AI indexers and wel…
We already know the solution: One well-behaved, shared scraper could serve all of the AI companies simultaneously. The problem is that they're not doing it.
Earlier quoted context omitted.
That's sorta what MetaBrainz did - they offer their whole DB as a single tarball dump, much like what Wikipedia does. I downloaded it in the order of an hour; if I need a MusicBrainz lookup, I just do a local query. For this strategy to work, people need to actually use the DB dumps instead of just defaulting to scraping. Unfortunately scraping is trivially easy, particularly now that AI code assistants can write a w…
I mean this AI data scrapper would need to scan and fetch billions of website why would they even care over 1 single website ??? You expect instiution to care out of billions website they must scrape daily
Metabrainz is a great resource -- I wrote about them a few years ago here: https://www.eff.org/deeplinks/2021/06/organizing-public-inte... There's something important here in that a public good like Metabrainz would be fine with the AI bots picking up their content -- they're just doing it in a frustratingly inefficient way. It's a co-ordination problem: Metabrainz assumes good intent from bots, and has to lock down…
I'm not sure why you're personifying what is almost certainly a script that fetches documents, parses all the links in them, and then recursively fetches all of those.
When we say "AI scraper" we're describing a crawler controlled by an AI company indiscriminately crawling the web, not a literal AI reading and reasoning about each page... I'm surprised this needs to be said.
Someone convinced me last time[0] that these aren't the well-known scrapers we know but other actors. We wouldn't be able to tell, certainly. I'd like to help the scrapers be better about reading my site, but I get why they aren't. I wish there were an established protocol for this. Say a $site/.well-known/machine-readable.json that instructs you on a handful of established software or allows pointing to an appropria…
> To archive Metabrainz there is no way but to browse the pages slowly page-by-page. There's no machine-communicable way that suggests an alternative. Why does there have to be a "machine-communicable way"? If these developers cared about such things they would spend 20 seconds looking at this page. It's literally one of the first links when you Google "metabrainz" https://metabrainz.org/datasets
Metabrainz is a great resource -- I wrote about them a few years ago here: https://www.eff.org/deeplinks/2021/06/organizing-public-inte... There's something important here in that a public good like Metabrainz would be fine with the AI bots picking up their content -- they're just doing it in a frustratingly inefficient way. It's a co-ordination problem: Metabrainz assumes good intent from bots, and has to lock down…
> they assume that the website is adversarially "hiding" its content. They won't believe a random site when it says "Look, stop hitting our API, you can pick all of this data in one go, over in this gzipped tar file." I'm not sure why you're personifying what is almost certainly a script that fetches documents, parses all the links in them, and then recursively fetches all of those. When we say "AI scraper" we're des…
Nowadays people complain about AI scrapers with the same vain as they complained about search indexers a way back when. Just a few years later, people had stopped caring too much about storage access and bandwidth, and started begging search engines to visit their websites. Every trick on the planet Earth, SEO optimization, etc. Looking forward to the time when everybody suddenly starts to embrace AI indexers and wel…
Earlier quoted context omitted.
They should advertise that. I pretty much reflexively avoid any mention of AI in interfaces because they usually mean "we're sending this all to openthropigoogosoft so I hope you don't have any secrets."
> They should advertise that They did Previews can optionally include AI-generated key points, which are processed on your device to protect your privacy. https://www.firefox.com/en-US/firefox/142.0/releasenotes/ I'll also add that if you go to the Labs page (in settings) you can enable another local model to semantically search your history
The SQLite team faced a similar problem last year, and Richard Hipp (the creator of SQLite) made almost the same comment: "The malefactor behind this attack could just clone the whole SQLite source repository and search all the content on his own machine, at his leisure. But no: Being evil, the culprit feels compelled to ruin it for everyone else. This is why you don't get to keep nice things...." https://sqlite.org/…
[flagged]
(1) root@gentoo-server ~ # egrep 'openai|claude' -c /var/log/lighttpd/access.log
8537094