Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We can't have nice things because of AI scrapers

blog.metabrainz.org

Post reply on HN
Print view View original

71–80 of 279 posts

Re: We can't have nice things because of AI scrapers

#71

Post by falloutx »

acatton wrote: post #44
falloutx wrote: post #33

Its not just AI scrappers doing it by themselves but now users are also being trained to put the link in the claude chat/chat gpt and ask it to summarise it. And off course that would show up on the website end as a scraper. In fact firefox now allows you to preview the link and get key points without ever going to the link[1] [1] https://imgur.com/a/3E17Dts

> In fact firefox now allows you to preview the link and get key points without ever going to the link[1] > [1] https://imgur.com/a/3E17Dts This is generated on device with llama.cpp compiled to webassembly (aka wllama) and running SmolLM2-360M. [1] How is this different from the user clicking on the link? In the end, your local firefox will fetch the link in order to summarize it, the same way you would have followe…

I actually didnt know it was using a local model and that it fetches it locally.

Re: We can't have nice things because of AI scrapers

#72

Post by gleenn »

VTimofeenko wrote: post #58
jacksnipe wrote: post #30

Earlier quoted context omitted.

It’s not great, but you could add it to the body of a 429 response.

Genuinely curious: do programs read bodies of 429 responses? In the code bases that I have seen, 429 is not read beyond the code itself

Almost certainly not by default, certainly not in any of the http libs I have used

Re: We can't have nice things because of AI scrapers

#73

Post by Afforess »

squigz wrote: post #62
randyl wrote: post #57

The SQLite team faced a similar problem last year, and Richard Hipp (the creator of SQLite) made almost the same comment: "The malefactor behind this attack could just clone the whole SQLite source repository and search all the content on his own machine, at his leisure. But no: Being evil, the culprit feels compelled to ruin it for everyone else. This is why you don't get to keep nice things...." https://sqlite.org/…

[flagged]

Sounds like you have zero empathy for the real costs AI is driving and feelings that this creates for website owners. How about you pony up and pay for your scraping?

Re: We can't have nice things because of AI scrapers

#74

Post by deathanatos »

jakelazaroff wrote: post #65
VTimofeenko wrote: post #58

Earlier quoted context omitted.

Genuinely curious: do programs read bodies of 429 responses? In the code bases that I have seen, 429 is not read beyond the code itself

Sometimes! The server can also send a retry-after header to indicate when the client is allowed to request the resource again: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...

… which isn't part of the body of a 429…

Re: We can't have nice things because of AI scrapers

#75

Post by ericd »

acatton wrote: post #44
falloutx wrote: post #33

Its not just AI scrappers doing it by themselves but now users are also being trained to put the link in the claude chat/chat gpt and ask it to summarise it. And off course that would show up on the website end as a scraper. In fact firefox now allows you to preview the link and get key points without ever going to the link[1] [1] https://imgur.com/a/3E17Dts

> In fact firefox now allows you to preview the link and get key points without ever going to the link[1] > [1] https://imgur.com/a/3E17Dts This is generated on device with llama.cpp compiled to webassembly (aka wllama) and running SmolLM2-360M. [1] How is this different from the user clicking on the link? In the end, your local firefox will fetch the link in order to summarize it, the same way you would have followe…

That’s awesome :-)

Like, can we all take a step back and marvel that freaking wasm can do things that 10 years ago were firmly in the realm of sci-fi?

I hope they’ll extend that sort of thing to help filter out the parts of the dom that represent attention grabbing stuff that isn’t quite an ad, but is still off topic/not useful for what I’m working on at the moment (and still keep the relevant links).

Re: We can't have nice things because of AI scrapers

#76

Post by Aurornis »

bitbasher wrote: post #26
SchemaLoad wrote: post #2

Cloudflare has a service for this now that will detect AI scrapers and send them to a tarpit of infinite AI generated nonsense pages.

Wow, so to prevent AI scrapers from harvesting my data I need to send all of my traffic through a third party company that gets to decide who gets to view my content. Great idea!

You don’t need to do anything. You can use any number of solutions or roll your own.

Someone shared an alternative. Must everything in AI threads be so negative and condescending?

Re: We can't have nice things because of AI scrapers

#77

Post by jjgreen »

kpcyrd wrote: post #54
tommek4077 wrote: post #43

How do they get overloaded? Is the website too slow? I have a quite big wiki online and barely see any impact from bots.

The API seems to be written in Perl: https://github.com/metabrainz/musicbrainz-server

Time for a vinyl-style Perl revival ...

Re: We can't have nice things because of AI scrapers

#78

Post by lysace »

randyl wrote: post #57

The SQLite team faced a similar problem last year, and Richard Hipp (the creator of SQLite) made almost the same comment: "The malefactor behind this attack could just clone the whole SQLite source repository and search all the content on his own machine, at his leisure. But no: Being evil, the culprit feels compelled to ruin it for everyone else. This is why you don't get to keep nice things...." https://sqlite.org/…

[flagged]

Re: We can't have nice things because of AI scrapers

#79

Post by squigz »

TeMPOraL wrote: post #63
squigz wrote: post #32

Earlier quoted context omitted.

The mechanism is putting some text that points to the downloads.

So perhaps it's time to standardize that .

I'm not entirely sure why people think more standards are the way forward. The scrapers apparently don't listen to the already-established standards. What makes one think they would suddenly start if we add another one or two?
Post reply on HN