Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We can't have nice things because of AI scrapers

blog.metabrainz.org

Post reply on HN
Print view View original

271–279 of 279 posts

Re: We can't have nice things because of AI scrapers

#271

Post by switz »

tommek4077 wrote: post #112
switz wrote: post #55

Earlier quoted context omitted.

If you have a lot of pages, AI bots will scrape every single one on a loop - wiki's generally don't have anywhere near the number of pages as an incremented entity primary id. I have a few million pages on a tiny website and it gets hammered by AI bots all day long. I can handle it, but it's a nuisance and they're basically just scraping garbage (statistics pages of historical matches or user pages that have essentia…

How many requests per second do you get? I also see a lot of bot traffic but nowhere near to hit the servers significantly, and i render most stuff on the server directly.

Around a hundred per second at peak. Even though my server can handle it just fine, it muddies up the logs and observability for something I genuinely do not care about at all. I only care about seeing real users' experience. It's just noise.

Re: We can't have nice things because of AI scrapers

#272

Post by wtetzner »

crazygringo wrote: post #190
wtetzner wrote: post #175

Earlier quoted context omitted.

Isn't the point of AI that it's good at understanding content written for humans? Why can't the scrapers run the homepage through an LLM to detect that? I'm also not sure why we should be prioritizing the needs of scraper writers over human users and site operators.

How is passing a site's homepage to an LLM supposed to make it develop a custom crawler?

It's not, the crawler would use the LLM to read the contents of the first page to dynamically determine the best way to capture the data (e.g. the zip file from TFA).

Re: We can't have nice things because of AI scrapers

#273

Post by mr_monkey »

8note wrote: post #170
nostrademons wrote: post #17

Earlier quoted context omitted.

That's sorta what MetaBrainz did - they offer their whole DB as a single tarball dump, much like what Wikipedia does. I downloaded it in the order of an hour; if I need a MusicBrainz lookup, I just do a local query. For this strategy to work, people need to actually use the DB dumps instead of just defaulting to scraping. Unfortunately scraping is trivially easy, particularly now that AI code assistants can write a w…

the obvious thing would be to take down their website and only have the DB dump. if thats the useful thing, it doesnt need the wrapper

Ah, yes, if the mafia comes to threaten your business for a protection racket, just abandon your shop, simple !

Re: We can't have nice things because of AI scrapers

#274

Post by mr_monkey »

nottorp wrote: post #211

Okay, it's been established that "AI" crawlers are a pest. One of the reasons being that they don't actually run any "AI", that would be too expensive. You can't ban by user agent because that will only catch the few crawlers that are actually honest about it. Aren't there rate limiting solutions built into at least some web servers? At least if you control your own web server, can't you do it through some reverse pr…

MetaBrainz ended up rolling out their own tool, none of the other common suggestions being quite appropriate.

Re: We can't have nice things because of AI scrapers

#275

Post by mr_monkey »

squigz wrote: post #3

> The /metadata/lookup API endpoints (GET and POST versions) now require the caller to send an Authorization token in order for this endpoint to work. > The ListenBrainz Labs API endpoints for mbid-mapping, mbid-mapping-release and mbid-mapping-explain have been removed. Those were always intended for debugging purposes and will also soon be replaced with a new endpoints for our upcoming improved mapper. > LB Radio w…

We can't have a free internet that does not demand identification and data collection as a price to pay.

Re: We can't have nice things because of AI scrapers

#276

Post by mr_monkey »

zx8080 wrote: post #35

Nothing prevents scraper from creating a free account and sending auth token in API requests. I'm not saying the API changes are pointless, but still, what's the catch?

Those botnets are hitting random endpoints thousands of times a minute. The problem is that each time they switch to a different residential IP so that they are untraceable. That's the frustrating part: not only do they not play by the rules, but they use advanced methods to obfuscate and bypass any protections. That probably costs them a fair amount too, all that to access free data they can download as a tar file...

They won't be able to create thousands of API keys a minute, and if they reuse the keys they'll very easily be identified and blocked.

Re: We can't have nice things because of AI scrapers

#277

Post by mr_monkey »

smallerfish wrote: post #14

Bear in mind that some portion of this could be human directed research. I'm doing a research project right now with 1000 things that I'm building profiles on; to build a full profile requires an agent to do somewhere around 100 different site lookups. Where APIs exist, I've registered API keys and had the agent write a script to query the data in that manner, but that required me to be deliberate about it. Non techn…

This is not that, it's thousand of hits a minute on random endpoints, basically scraping everything all the time.

Re: We can't have nice things because of AI scrapers

#278

Post by tux »

tux wrote: post #180
Cadwhisker wrote: post #174

Earlier quoted context omitted.

I took a look at the https://tvnfo.com/ site and I have no idea what's behind the donation wall. Can I suggest you have a single page which explains or demonstrates the content, or there's no reason for "new" people to want to donate to get access.

Yeah i’ll have something up soon :-)

Okay, see front page I added some screenshots :-)
Post reply on HN