Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We can't have nice things because of AI scrapers

blog.metabrainz.org

Post reply on HN
Print view View original

221–230 of 279 posts

Re: We can't have nice things because of AI scrapers

#221

Post by TitaRusell »

dannersy wrote: post #207
Aurornis wrote: post #76

Earlier quoted context omitted.

You don’t need to do anything. You can use any number of solutions or roll your own. Someone shared an alternative. Must everything in AI threads be so negative and condescending?

Yes, they could roll their own, but you have no issues with this being necessary? I think the attitude of "just deal with it" is far more negative than someone expressing they are upset with the state of the internet, its controllers, and its abusers.

There's trillions invested in AI. Don't expect any introspective insight or criticism about it.

Re: We can't have nice things because of AI scrapers

#222

Post by tonyedgecombe »

Jeremy1026 wrote: post #181

I sysadmin my kids' PTA website. OpenAI was scraping it recently. I saw it looking at the event calendar, request after request to random days. I saw years 1000 through 3000 scroll by. I changed the response to their user agent to an access denied, but it still took about 4 hours for them to stop.

I wonder who wrote a crawler like that. Oh wait ...

Re: We can't have nice things because of AI scrapers

#224

Post by mlrtime »

everybodyknows wrote: post #166
bodantogat wrote: post #19

I feel the pain — it’s very difficult to detect many of the less ethical scrapers. They use residential IP pools, rotate IPs, and provide valid user agents.

> residential IP pools So, is this a new profit center for sleazeball household ISPs?

"free" vpns do this.

Re: We can't have nice things because of AI scrapers

#226

Post by jijijijij »

akuyou wrote: post #138

AI is destroying the free internet along with everything else My web host suspended my website account last week due to a sudden large volume of requests to it - effectively punishing me for being scraped by bots. I've had to move to a new host to get back up, but what hope does the little guy have? it's like GPU and ram prices, it doesn't matter if I pay 10x 100x or 1000x more than I did, the AI companies have infin…

My prediction: AI is the deathblow to IPv6 adoption for the wider web, since blocklists only really work with IPv4. Increasing VPN usage making user tracking and heuristics difficult, AI scrapers stealing appropriated human content and AI spam poisoning its exploitation, not to mention tech monopolization and centralization, the limitations of IPv4 are suddenly becoming an asset and incentives for IPv6 support are zero.

On the plus side, we probably got all of IPv6 to build an alternative, non-commercial, better web or whatever network, if we act quickly before routing support is vanishing. IPv6-only housing is cheap, things don’t need to work out. There could be the IPv4-enforced corpo world within walls, and IPv6-enabled wild wide wonderlands.

Re: We can't have nice things because of AI scrapers

#227

Post by squigz »

DrewADesign wrote: post #164
godelski wrote: post #144

Earlier quoted context omitted.

> They should advertise that They did Previews can optionally include AI-generated key points, which are processed on your device to protect your privacy. https://www.firefox.com/en-US/firefox/142.0/releasenotes/ I'll also add that if you go to the Labs page (in settings) you can enable another local model to semantically search your history

Ok, they should advertise it more .

It literally tells you this in the modal the first time you try this feature.

How else do you want them to advertise it?

Re: We can't have nice things because of AI scrapers

#228

Post by aembleton »

zahlman wrote: post #202
lelanthran wrote: post #199

Earlier quoted context omitted.

I'm interested in that service and how it works. Link?

It is https://github.com/silentsoft/hits . It works by loading an SVG "shield" file (like the ones you see at the top of GitHub readmes all the time) from their server from a unique URL (you just choose one when you write/render your HTML). The server, implemented in Java, just counts hits to each URL in a database and sends back the corresponding SVG data. There's also a mini dashboard website where you can check ba…

They're probably not downloading every svg each time they scrape the site. Probably focused on scraping the text.

Re: We can't have nice things because of AI scrapers

#229

Post by touisteur »

jijijijij wrote: post #226
akuyou wrote: post #138

AI is destroying the free internet along with everything else My web host suspended my website account last week due to a sudden large volume of requests to it - effectively punishing me for being scraped by bots. I've had to move to a new host to get back up, but what hope does the little guy have? it's like GPU and ram prices, it doesn't matter if I pay 10x 100x or 1000x more than I did, the AI companies have infin…

My prediction: AI is the deathblow to IPv6 adoption for the wider web, since blocklists only really work with IPv4. Increasing VPN usage making user tracking and heuristics difficult, AI scrapers stealing appropriated human content and AI spam poisoning its exploitation, not to mention tech monopolization and centralization, the limitations of IPv4 are suddenly becoming an asset and incentives for IPv6 support are ze…

There are ways to build blocklists for IPv6. I saw (used) once bloom filters for this. Inspired by some papers from the 2000s, this one in 2009 https://www.nokia.com/bell-labs/publications-and-media/publi...

Re: We can't have nice things because of AI scrapers

#230

Post by immibis »

Afforess wrote: post #73
squigz wrote: post #62

Earlier quoted context omitted.

[flagged]

Sounds like you have zero empathy for the real costs AI is driving and feelings that this creates for website owners. How about you pony up and pay for your scraping?

There's zero evidence any of this is related to AI btw
Post reply on HN