Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We can't have nice things because of AI scrapers

blog.metabrainz.org

Post reply on HN
Print view View original

261–270 of 279 posts

Re: We can't have nice things because of AI scrapers

#261

Post by bs7280 »

dannersy wrote: post #207
Aurornis wrote: post #76

Earlier quoted context omitted.

You don’t need to do anything. You can use any number of solutions or roll your own. Someone shared an alternative. Must everything in AI threads be so negative and condescending?

Yes, they could roll their own, but you have no issues with this being necessary? I think the attitude of "just deal with it" is far more negative than someone expressing they are upset with the state of the internet, its controllers, and its abusers.

This is like saying "lets just get rid of all the guns" to solve gun violence and gun crime in the USA. The cat is out of the bag and no one can put it back. We live in a different world now and we have to figure it out.

Re: We can't have nice things because of AI scrapers

#262

Post by DanOpcode »

kermatt wrote: post #256
DanOpcode wrote: post #232

I don't get why the AI scrapers need to scrape the same sites and pages over and over again.

Because the scrapers are poorly written. Efficiency is not a concern for them.

Feels like parasites who are killing their hosts by DDOSing them to death

Re: We can't have nice things because of AI scrapers

#263

Post by adrianwaj »

tdeck wrote: post #241
adrianwaj wrote: post #172

Look no further than x402 micropayments as both the solution and opportunity here. And then a way to return a portion to humans. These AI companies are loaded too (maybe not the long-tail as yet) and the crypto ecosystem is mature. Come one, come all. Make money. Need a Wordpress plugin to start the ball rolling and provide ping endpoints for the AI companies to leach from. They can pay to get those pings too. Give t…

So your local diner used to have napkin dispensers on every table full of napkins. Then some people started coming in and just taking all the napkins out when they left the restaurant. Now the diner is going broke paying for napkins so they stop stocking them. Seems like we can't have nice things :(. But wait! Great solution! We created a micropayment system. Sign up on your phone and use this QR code to pay 1 cent t…

The dispenser should be placed next to the cashier. There could also be a smaller amount in the dispenser in case someone grabs all of them.

The micropayment way is more like a vending machine. The issue is pricing, charge too much and a clone machine will pop up that will be cheaper for the buyer but still make the seller money. Charge too little -- well you'd still be making more than now but there could be a lot of money left on the table.

I actually think payments should be variable. People on this thread rant about having a designated way to allow for a big download, how about a designated way to make a payment (even if voluntary) ?

x402.txt / crypto.txt / coins.txt / authors.txt / tips.txt / supporters.txt

Then give people a way to "show off their generosity."

Re: We can't have nice things because of AI scrapers

#264

Post by jacksnipe »

VTimofeenko wrote: post #58
jacksnipe wrote: post #30

Earlier quoted context omitted.

It’s not great, but you could add it to the body of a 429 response.

Genuinely curious: do programs read bodies of 429 responses? In the code bases that I have seen, 429 is not read beyond the code itself

Up until very recently I would have said definitely not, but we're talking about LLM scrapers, who knows how much they've got crammed into their context windows.

Re: We can't have nice things because of AI scrapers

#265

Post by olyjohn »

chrismorgan wrote: post #231
olyjohn wrote: post #212

Earlier quoted context omitted.

How does a human abuse a website, and how does a CAPTCHA stop a human from abusing a website when it's designed to let a human in? If it doesn't stop humans from abusing the site, then it must stop... computers from abusing the site. And it stops computers by using the CAPTCHA to tell apart a human and a computer? Am I wrong here?

I can tell you don’t live in a network Cloudflare hates.

I get hit with them all the time. That doesn't answer how does a human abuse a website? Hitting F5 on the keyboard 1000 times?

Re: We can't have nice things because of AI scrapers

#267

Post by chrismorgan »

olyjohn wrote: post #265
chrismorgan wrote: post #231

Earlier quoted context omitted.

I can tell you don’t live in a network Cloudflare hates.

I get hit with them all the time. That doesn't answer how does a human abuse a website? Hitting F5 on the keyboard 1000 times?

Abuse isn’t just about inducing CPU or network traffic load (in fact I doubt that was even considered when CAPTCHA was first invented). It’s spam, fraud, that kind of thing. A lot of it is bot-perpetrated, but quite a bit is by humans too.

Re: We can't have nice things because of AI scrapers

#268

Post by jwe »

tonyedgecombe wrote: post #222
Jeremy1026 wrote: post #181

I sysadmin my kids' PTA website. OpenAI was scraping it recently. I saw it looking at the event calendar, request after request to random days. I saw years 1000 through 3000 scroll by. I changed the response to their user agent to an access denied, but it still took about 4 hours for them to stop.

I wonder who wrote a crawler like that. Oh wait ...

I can't follow your thought there. Who wrote it?

Re: We can't have nice things because of AI scrapers

#269

Post by account42 »

switz wrote: post #55
tommek4077 wrote: post #43

How do they get overloaded? Is the website too slow? I have a quite big wiki online and barely see any impact from bots.

If you have a lot of pages, AI bots will scrape every single one on a loop - wiki's generally don't have anywhere near the number of pages as an incremented entity primary id. I have a few million pages on a tiny website and it gets hammered by AI bots all day long. I can handle it, but it's a nuisance and they're basically just scraping garbage (statistics pages of historical matches or user pages that have essentia…

Even moderately sized wikis have a huge number of different page versions which can all be accessed individually.
Post reply on HN