We can't have nice things because of AI scrapers
blog.metabrainz.org
We can't have nice things because of AI scrapers
1–10 of 279 posts
Re: We can't have nice things because of AI scrapers
#2Re: We can't have nice things because of AI scrapers
#3> The ListenBrainz Labs API endpoints for mbid-mapping, mbid-mapping-release and mbid-mapping-explain have been removed. Those were always intended for debugging purposes and will also soon be replaced with a new endpoints for our upcoming improved mapper.
> LB Radio will now require users to be logged in to use it (and API endpoint users will need to send the Authorization header). The error message for logged in users is a bit clunky at the moment; we’ll fix this once we’ve finished the work for this year’s Year in Music.
Seems reasonable and no big deal at all. I'm not entirely sure what "nice things" we can't have because of this. Unauthenticated APIs?
Re: We can't have nice things because of AI scrapers
#4(Blocking Chinese IP ranges with the help of some geoip db helps a lot in the short term. Azure as a whole is the second largest source of pure idiocy.)
Re: We can't have nice things because of AI scrapers
#5Cloudflare has a service for this now that will detect AI scrapers and send them to a tarpit of infinite AI generated nonsense pages.
Re: We can't have nice things because of AI scrapers
#6Cloudflare has a service for this now that will detect AI scrapers and send them to a tarpit of infinite AI generated nonsense pages.
Do you have a link to that?
Re: We can't have nice things because of AI scrapers
#7Re: We can't have nice things because of AI scrapers
#8i don't want people's servers to be pegged at 100% because a stupid dfs scraper is exhaustively traversing their search facets, but i also want the web to remain scrapable by ordinary people, or rather go back to how readily scrapable it used to be before the invention of cloudflare
as a middle ground, perhaps we could agree on a new /.well-known/ path meant to contain links to timestamped data dumps?
Re: We can't have nice things because of AI scrapers
#9This is frustrating to watch. MetaBrainz is exactly the kind of project AI companies should be supporting—open data, community-maintained, freely available for download in bulk. Instead they’re: ∙ Ignoring robots.txt (the bare minimum web courtesy) ∙ Bypassing the provided bulk download (literally designed for this use case) ∙ Scraping page-by-page (inefficient for everyone) ∙ Overloading volunteer-run infrastructure…
Scraping page-by-page (inefficient for everyone)
you know what else is "(inefficient for everyone)"? posting the output instead of the promptRe: We can't have nice things because of AI scrapers
#10Mind you I take effort to not be burdensome by downloading only what I need and taking time between each request of a couple seconds, and the total data usage is low.
Ironically, I supposed you could call it "AI" what I'm using it for, but really it's just data analytics.