AI scrapers request commented scripts • Hacker News Board

Re: AI scrapers request commented scripts

#211

Post by bigbuppo » Sat, Nov 01, 2025, 5:29 PM UTC

michaelt wrote: post #43

stray wrote: post #40

Earlier quoted context omitted.

You require something the bot won't have that a human would. Anybody may watch the demo screen of an arcade game for free, but you have to insert a quarter to play — and you can have even greater access with a key. > and you’ve explicitly left a sign saying ‘you are not welcome here’ And the sign said "Long-haired freaky people Need not apply" So I tucked my hair up under my hat And I went in to ask him why He said,…

> You require something the bot won't have that a human would. Is this why the “open web” is showing me a captcha or two, along with their cookie banner and newsletter pop up these days?

Up until people started making a big stink about CAPTCHAs being used for unpaid labor at scale, uh, well they had two purposes.

Re: AI scrapers request commented scripts

#212

Post by bigbuppo » Sat, Nov 01, 2025, 5:30 PM UTC

bakql wrote: post #26

hsbauauvhabzb wrote: post #24

Earlier quoted context omitted.

How else do you tell the bot you do not wish to be scraped? Your analogy is lacking - you didn’t order a package, you never wanted a package, and the postman is taking something, not leaving it, and you’ve explicitly left a sign saying ‘you are not welcome here’.

Stop your http server if you do not wish to receive http requests.

Ah yes, and unplug the mail server to stop all spam. Great idea!

Re: AI scrapers request commented scripts

#213

Post by bigbuppo » Sat, Nov 01, 2025, 5:31 PM UTC

davsti4 wrote: post #32

hsbauauvhabzb wrote: post #24

Earlier quoted context omitted.

How else do you tell the bot you do not wish to be scraped? Your analogy is lacking - you didn’t order a package, you never wanted a package, and the postman is taking something, not leaving it, and you’ve explicitly left a sign saying ‘you are not welcome here’.

Its simple, and I'll quote myself - "robots.txt isn't the law".

Violating norms makes you an abusive jerk at best.

Re: AI scrapers request commented scripts

#214

Post by bigbuppo » Sat, Nov 01, 2025, 5:33 PM UTC

nkrisc wrote: post #194

kelnos wrote: post #68

Earlier quoted context omitted.

But I do want my content accessible to "just anyone", as long as they are humans. I don't want it accessible to bots. You are free to say "well, there is no mechanism to do that", and I would agree with you. That's the problem!

Even abusive crawlers and scrapers are acting as agents of real humans, just as your browser is acting as your agent. I don't even know how you could reliably draw a reasonable line in the sand between the two without putting some group of people on the wrong side of the line. I suppose the ultimate solution would be browsers and operating systems and hardware manufacturers co-operating to implement some system that…

Or the AI people could just stop being abusive jerks. That's an even easier solution.

Re: AI scrapers request commented scripts

#215

Post by bigbuppo » Sat, Nov 01, 2025, 5:34 PM UTC

Anamon wrote: post #185

1gn15 wrote: post #125

Earlier quoted context omitted.

What the hell? That is incredibly discriminatory. Fuck off. I support those that counter those discriminatory mechanisms.

Discriminatory against bots? That doesn't even make any sense.

They probably have stock options.

Re: AI scrapers request commented scripts

#216

Post by bigbuppo » Sat, Nov 01, 2025, 5:47 PM UTC

whimsicalism wrote: post #31

Calavar wrote: post #22

Earlier quoted context omitted.

I agree. It always surprises me when people are indignant about scrapers ignoring robots.txt and throw around words like "theft" and "abuse." robots.txt is a polite request to please not scrape these pages because it's probably not going to be productive. It was never meant to be a binding agreement, otherwise there would be a stricter protocol around it. It's kind of like leaving a note for the deliveryman saying pl…

There's an evolving morality around the internet that is very, very different from the pseudo-libertarian rule of the jungle I was raised with. Interesting to see things change.

You're very much wrong. Two of the key tennets of libertarianism is that your rights end where my nose begins and the respect of property rights . Your AI bot is causing problems for me, then you should be compensating me for the damage or other expense you caused. But the AI bros think they should be able to take anything they want whenever they want without compensation, and they'll use every single shady behavior they can to make that happen. In other words, they're robber barrons.

Re: AI scrapers request commented scripts

#217

Post by DoctorOetker » Sat, Nov 01, 2025, 5:48 PM UTC

kelnos wrote: post #63

Calavar wrote: post #22

Earlier quoted context omitted.

I agree. It always surprises me when people are indignant about scrapers ignoring robots.txt and throw around words like "theft" and "abuse." robots.txt is a polite request to please not scrape these pages because it's probably not going to be productive. It was never meant to be a binding agreement, otherwise there would be a stricter protocol around it. It's kind of like leaving a note for the deliveryman saying pl…

> robots.txt is a polite request to please not scrape these pages People who ignore polite requests are assholes, and we are well within our rights to complain about them. I agree that "theft" is too strong (though I think you might be presenting a straw man there), but "abuse" can be perfectly apt: a crawler hammering a server, requesting the same pages over and over, absolutely is abuse. > Likewise if enforcing a r…

Whenever one forms a sentence, it is worthwhile to try to form a sentence that you believe to be generally true.

If someone politely requests you to suck their genitalia, and you ignore that request, does that make you an asshole?

Re: AI scrapers request commented scripts

#218

Post by tremon » Sat, Nov 01, 2025, 6:10 PM UTC

ryandrake wrote: post #82

jraph wrote: post #64

Earlier quoted context omitted.

Any analogy is flawed and you can kill most analogies very fast. They are meant to illustrate a point hopefully efficiently, not to be mathematically true. They are not to everyone's taste, me included in most cases. They are mostly fine as long as they are not used to make a point, but only to illustrate it. I agree with this criticism of this analogy, I actually had this flaw in mind from the start. There are other…

People need to have a better mental model of what it means to host a public web site, and what they are actually doing when they run the web server and point it at a directory of files. They're not just serving those files to customers. They're not just serving them to members. They're not just serving them to human beings. They're not even necessarily serving files to web browsers. They're serving files to every IP…

The CFAA wants to have a word. The fact that a server responds with a 200 OK has no bearing on the legality of your request, there's plenty of precedent by now.

Re: AI scrapers request commented scripts

#219

Post by 9rx » Sat, Nov 01, 2025, 6:31 PM UTC

bigbuppo wrote: post #214

nkrisc wrote: post #194

Earlier quoted context omitted.

Even abusive crawlers and scrapers are acting as agents of real humans, just as your browser is acting as your agent. I don't even know how you could reliably draw a reasonable line in the sand between the two without putting some group of people on the wrong side of the line. I suppose the ultimate solution would be browsers and operating systems and hardware manufacturers co-operating to implement some system that…

Or the AI people could just stop being abusive jerks. That's an even easier solution.

While that is probably good advice in general, the earlier commenter wanted even the abusive jerks to have access to his content.

He just doesn't want tools humans use to access content to be used in association with his content.

What he failed to realize is that if you eliminate the tools, the human cannot access the content anyway. They don't have the proper biological interfaces. Had he realized that, he'd have come to notice that simply turning off his server fully satisfies the constraints.

Re: AI scrapers request commented scripts

#220

Post by nkrisc » Sat, Nov 01, 2025, 7:47 PM UTC

bigbuppo wrote: post #214

nkrisc wrote: post #194

Earlier quoted context omitted.

Even abusive crawlers and scrapers are acting as agents of real humans, just as your browser is acting as your agent. I don't even know how you could reliably draw a reasonable line in the sand between the two without putting some group of people on the wrong side of the line. I suppose the ultimate solution would be browsers and operating systems and hardware manufacturers co-operating to implement some system that…

Or the AI people could just stop being abusive jerks. That's an even easier solution.

That would be easier. Too bad it won't ever happen.