Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We can't have nice things because of AI scrapers

blog.metabrainz.org

Post reply on HN
Print view View original

231–240 of 279 posts

Re: We can't have nice things because of AI scrapers

#231

Post by chrismorgan »

olyjohn wrote: post #212
chrismorgan wrote: post #186

Earlier quoted context omitted.

Telling humans and computers apart was never the purpose of CAPTCHAs, only how they initially worked. The name has been a complete misnomer for at least a decade now. Its actual purpose is, and has always been, abuse prevention. Has it been successful? Some yes, some no, and a lot of collateral damage. Its mode of operation now looks a lot like inscrutable blacklisting for some plus inconvenience and bad rate limitin…

How does a human abuse a website, and how does a CAPTCHA stop a human from abusing a website when it's designed to let a human in? If it doesn't stop humans from abusing the site, then it must stop... computers from abusing the site. And it stops computers by using the CAPTCHA to tell apart a human and a computer? Am I wrong here?

I can tell you don’t live in a network Cloudflare hates.

Re: We can't have nice things because of AI scrapers

#233

Post by jijijijij »

touisteur wrote: post #229
jijijijij wrote: post #226

Earlier quoted context omitted.

My prediction: AI is the deathblow to IPv6 adoption for the wider web, since blocklists only really work with IPv4. Increasing VPN usage making user tracking and heuristics difficult, AI scrapers stealing appropriated human content and AI spam poisoning its exploitation, not to mention tech monopolization and centralization, the limitations of IPv4 are suddenly becoming an asset and incentives for IPv6 support are ze…

There are ways to build blocklists for IPv6. I saw (used) once bloom filters for this. Inspired by some papers from the 2000s, this one in 2009 https://www.nokia.com/bell-labs/publications-and-media/publi...

The point isn't the technical inability to block particular IPv6 addresses efficiently, but anticipating abuse potential by IP. You can change IPv6 addresses freely compared to IPv4. With IPv4 it's easy to determine, if you are dealing with a residential IP or VPN. No heuristics or analysis needed. IPv4 addresses are blocked preemptively, that's not really a thing for IPv6. Eg. VPN providers wouldn't have static endpoint addresses with IPv6. So you may be able to limit spontaneous abuse such as DDoS attacks, but it's a lot harder to filter technically legitimate traffic, which is merely unwanted for your data aggregation.

Re: We can't have nice things because of AI scrapers

#234

Post by dpkirchner »

zimpenfish wrote: post #214
dpkirchner wrote: post #143

Earlier quoted context omitted.

Nor Safari on iOS.

Works fine on my iOS Safari - maybe there's some extension that's tickling it just the wrong way?

It still fails with all of my extensions disabled (wipr, privacy redirect). I just get a download dialog. I don't know what the HTTP status code is, however.

I found a flagged HN submission about it and it has just about the same result for me and for others. My first tap failed in a weird way (showed some text then redirected quickly to its git repo) and all subsequent taps trigger a download.

https://news.ycombinator.com/item?id=44538010

Re: We can't have nice things because of AI scrapers

#236

Post by saaaaaam »

arjie wrote: post #81
saaaaaam wrote: post #36

Earlier quoted context omitted.

As referenced in the article, there absolutely is an alternative. https://metabrainz.org/datasets Linked to from the homepage as “datasets”. I may be too broadly interpreting what you mean by “machine-communicable” in the context of AI scraping though.

Well, imagine the best case and that you're a cooperative bot writer who does not intend to harm website owners. Okay, so you follow robots.txt and all that. That's straightforward. But it's not like you're writing a "metabrainz crawler" and a "metafilter crawler" and a "wiki.roshangeorge.dev crawler". You're presumably trying to write a general Internet crawler. You encounter a site that is clearly a HTTP view into…

Right, yes, I see your point. I was thinking more from the point of view of "using AI to explore and then write custom scrapers where relevant" rather than just blanket scraping. But you're right - at the scale we're talking, it's presumably just blunt-force "point-and-go" scraping, rather than anything more nuanced.

The point you make about having some sort of indicator that scrapers can follow to scrape in an optimal way (or access a dump) makes a lot of sense for people who want their content to be ingested by AI.

Re: We can't have nice things because of AI scrapers

#237

Post by lmz »

jijijijij wrote: post #233
touisteur wrote: post #229

Earlier quoted context omitted.

There are ways to build blocklists for IPv6. I saw (used) once bloom filters for this. Inspired by some papers from the 2000s, this one in 2009 https://www.nokia.com/bell-labs/publications-and-media/publi...

The point isn't the technical inability to block particular IPv6 addresses efficiently, but anticipating abuse potential by IP. You can change IPv6 addresses freely compared to IPv4. With IPv4 it's easy to determine, if you are dealing with a residential IP or VPN. No heuristics or analysis needed. IPv4 addresses are blocked preemptively, that's not really a thing for IPv6. Eg. VPN providers wouldn't have static endp…

Is there anything against just blocking at the /48 level?

Re: We can't have nice things because of AI scrapers

#239

Post by jijijijij »

lmz wrote: post #237
jijijijij wrote: post #233

Earlier quoted context omitted.

The point isn't the technical inability to block particular IPv6 addresses efficiently, but anticipating abuse potential by IP. You can change IPv6 addresses freely compared to IPv4. With IPv4 it's easy to determine, if you are dealing with a residential IP or VPN. No heuristics or analysis needed. IPv4 addresses are blocked preemptively, that's not really a thing for IPv6. Eg. VPN providers wouldn't have static endp…

Is there anything against just blocking at the /48 level?

No, but subnets can't be as easily associated with unwanted traffic. If IPv6 gets blocked you just get another IP. A VPN or hosting provider can't simply rent, or god forbid buy IPv4 addresses and subnets, arbitrarily. The IPs they use are rather static and easy to discover. Rather trivial to block all them, preemptively. Residential IPv4 VPNs are not legal offerings and their use is limited. VPNs can fight traffic analysis, they can't fight preemptive IPv4 blocking.

See, it doesn't matter if it's somehow possible to control IPv6 traffic, factually, it is sooo much easier to control and observe IPv4. IPv6 adoption isn't going great at all and now there are new strong business incentives against it.

The direction we're moving right now isn't free intergalactic mesh networking, but holistic control and centralization by the tech oligarchy. IPv6 is good things... we can't have those.

Re: We can't have nice things because of AI scrapers

#240

Post by tdeck »

incompatible wrote: post #28
lysace wrote: post #4

At some point they must become more cost efficient by pure market economics mechanisms. That implies less load on sites. Much of the scraping that I see is still very dumb/repetative. Like Googlebot in like 2001. (Blocking Chinese IP ranges with the help of some geoip db helps a lot in the short term. Azure as a whole is the second largest source of pure idiocy.)

They seem to have so much bubble money at the moment that the cost of scraping is probably a rounding error in their pocket change.

So the cost of caching should be a rounding error as well. If The Internet Archive can afford to cache vast swathes of the web, then surely the big AI companies can do so.
Post reply on HN