Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI scrapers request commented scripts

cryptography.dog

Post reply on HN
Print view View original

61–70 of 234 posts

Re: AI scrapers request commented scripts

#61

Post by mxkopy »

Calavar wrote: post #22
bakql wrote: post #20

>These were scrapers, and they were most likely trying to non-consensually collect content for training LLMs. "Non-consensually", as if you had to ask for permission to perform a GET request to an open HTTP server. Yes, I know about weev. That was a travesty.

I agree. It always surprises me when people are indignant about scrapers ignoring robots.txt and throw around words like "theft" and "abuse." robots.txt is a polite request to please not scrape these pages because it's probably not going to be productive. It was never meant to be a binding agreement, otherwise there would be a stricter protocol around it. It's kind of like leaving a note for the deliveryman saying pl…

The metaphor doesn’t work. It’s not the security of the package that’s in question, but something like whether the delivery person is getting paid enough or whether you’re supporting them getting replaced by a robot. The issue is in the context, not the protocol.

Re: AI scrapers request commented scripts

#62

Post by ryandrake »

jraph wrote: post #58
jMyles wrote: post #50

Earlier quoted context omitted.

Sexual consent is sacred. This metaphor is in truly bad taste. When you return a response with a 200-series status code, you've granted consent. If you don't want to grant consent, change the logic of the server.

> When you return a response with a 200-series status code, you've granted consent. If you don't want to grant consent, change the logic of the server. "If you don't consent to me entering your house, change its logic so that picking the door's lock doesn't let me open the door" Yeah, well… As if the LLM scrappers didn't try anything under the sun like using millions of different residential IP to prevent admins from…

The locked door is a ridiculous analogy when it comes to the open web. Pretty much all "door" analogies are flawed, but sure let's imagine your web server has a door. If you want to actually lock the door, you're more than welcome to put an authentication gate around your content. A web server that accepts a GET request and replies 2xx is distinctly NOT "locked" in any way.

Re: AI scrapers request commented scripts

#63

Post by kelnos »

Calavar wrote: post #22
bakql wrote: post #20

>These were scrapers, and they were most likely trying to non-consensually collect content for training LLMs. "Non-consensually", as if you had to ask for permission to perform a GET request to an open HTTP server. Yes, I know about weev. That was a travesty.

I agree. It always surprises me when people are indignant about scrapers ignoring robots.txt and throw around words like "theft" and "abuse." robots.txt is a polite request to please not scrape these pages because it's probably not going to be productive. It was never meant to be a binding agreement, otherwise there would be a stricter protocol around it. It's kind of like leaving a note for the deliveryman saying pl…

> robots.txt is a polite request to please not scrape these pages

People who ignore polite requests are assholes, and we are well within our rights to complain about them.

I agree that "theft" is too strong (though I think you might be presenting a straw man there), but "abuse" can be perfectly apt: a crawler hammering a server, requesting the same pages over and over, absolutely is abuse.

> Likewise if enforcing a rule of no scraping is of utmost importance you need to require an API token or some other form of authentication before you serve the pages.

That's a shitty world that we shouldn't have to live in.

Re: AI scrapers request commented scripts

#64

Post by jraph »

ryandrake wrote: post #62
jraph wrote: post #58

Earlier quoted context omitted.

> When you return a response with a 200-series status code, you've granted consent. If you don't want to grant consent, change the logic of the server. "If you don't consent to me entering your house, change its logic so that picking the door's lock doesn't let me open the door" Yeah, well… As if the LLM scrappers didn't try anything under the sun like using millions of different residential IP to prevent admins from…

The locked door is a ridiculous analogy when it comes to the open web. Pretty much all "door" analogies are flawed, but sure let's imagine your web server has a door. If you want to actually lock the door, you're more than welcome to put an authentication gate around your content. A web server that accepts a GET request and replies 2xx is distinctly NOT "locked" in any way.

Any analogy is flawed and you can kill most analogies very fast. They are meant to illustrate a point hopefully efficiently, not to be mathematically true. They are not to everyone's taste, me included in most cases. They are mostly fine as long as they are not used to make a point, but only to illustrate it.

I agree with this criticism of this analogy, I actually had this flaw in mind from the start. There are other flaws I have in mind as well.

I have developed more without the analogy in the remaining of the comment. How about we focus on the crux of the matter?

> A web server that accepts a GET request and replies 2xx is distinctly NOT "locked" in any way

The point is that these scrappers use tricks so that it's difficult not to grant them access. What is unreasonable here is to think that 200 means consent, especially knowing about the tricks.

Edit:

> you're more than welcome to put an authentication gate around your content.

I don't want to. Adding auth so llm providers don't abuse my servers and the work I meant to share publicly is not a working solution.

Re: AI scrapers request commented scripts

#65

Post by kelnos »

Calavar wrote: post #27
hsbauauvhabzb wrote: post #24

Earlier quoted context omitted.

How else do you tell the bot you do not wish to be scraped? Your analogy is lacking - you didn’t order a package, you never wanted a package, and the postman is taking something, not leaving it, and you’ve explicitly left a sign saying ‘you are not welcome here’.

If you are serving web pages, you are soliciting GET requests, kind of like ordering a package is soliciting a delivery. "Taking" versus "giving" is neither here nor there for this discussion. The question is are you expressing a preference on etiquette versus a hard rule that must be followed. I personally believe robots.txt is the former, and I say that as someone who serves more pages than they scrape

> If you are serving web pages, you are soliciting GET requests

So what's the solution? How do I host a website that welcomes human visitors, but rejects all scrapers?

There is no mechanism! The best I can do is a cat-and-mouse arms race where I try to detect the traffic I don't want, and block it, while the people generating the traffic keep getting more sophisticated about hiding from my detection.

No, putting up a paywall is not a reasonable response to this.

> The question is are you expressing a preference on etiquette versus a hard rule that must be followed.

Well, there really aren't any hard rules that must be followed, because there are no enforcement mechanisms outside of going nuclear (requiring login). Everything is etiquette. And I agree that robots.txt is also etiquette, and it is super messed up that we tolerate "AI" companies stomping all over that etiquette.

Do we maybe want laws that say everyone must respect robots.txt? Maybe? But then people will just move their scrapers to a jurisdiction without those laws. And I'm sure someone could make the argument that robots.txt doesn't apply to them because they spoofed a browser user-agent (or another user-agent that a site explicitly allows). So perhaps we have a new mechanism, or new laws, or new... something.

But this all just highlights the point I'm making here: there is no reasonable mechanism (no, login pages and http auth don't count) for site owners to restrict access to their site based on these sorts of criteria. And that's a problem.

Re: AI scrapers request commented scripts

#66

Post by vaylian »

OhMeadhbh wrote: post #14

I blame modern CS programs that don't teach kids about parsing. The last time I looked at some scraping code, the dev was using regexes to "parse" html to find various references. Maybe that's a way to defend against bots that ignore robots.txt, include a reference to a Honeypot HTML file with garbage text, but include the link to it in a comment.

The people who do this type of scraping to feed their AI are probably also using AI to write their scraper.

Re: AI scrapers request commented scripts

#67

Post by kelnos »

jMyles wrote: post #55
munk-a wrote: post #41

Earlier quoted context omitted.

I disagree strongly here - though not from a technical perspective. There's absolutely a legal concept of making your work available for viewing without making it available for copying and AI scraping (while we can technically phrase it as just viewing a bunch of times) is effectively copying. Lets say a large art hosting site realizes how damaging AI training on their data can be - should they respond by adding a pa…

> There's absolutely a legal concept of making your work available for viewing without making it available for copying This "legal concept" is enforceable through legacy systems of police and violence. The internet does not recognize it. How much more obvious can this get? If we stumble down the path of attempting to apply this legal framework, won't some jurisdiction arise with no IP protections whatsoever and just…

> legacy systems of police and violence

You use "legacy" as if these systems are obsolete and on their way out. They're not. They're here to stay, and will remain dominant, for better or worse. Calling them "legacy" feels a bit childish, as if you're trying to ignore reality and base arguments on your preferred vision of how things should be.

> The internet does not recognize it.

Sure it does. Not universally, but there are a lot of things governments and law enforcement can do to control what people see and do on the internet.

> If we stumble down the path of attempting to apply this legal framework, won't some jurisdiction arise with no IP protections whatsoever and just come to completely dominate the entire economy of the internet?

No, of course not, that's silly. That only really works on the margins. Any other country would immediately slap economic sanctions on that free-for-all jurisdiction and cripple them. If that fails, there's always a military response they can resort to.

> If I can spin up a server in copyleftistan with a complete copy of every album and film ever made, available for free download, why would users in copyrightistan use the locked down services of their domestic economy?

Because the governments of all the copyrightistans will block all traffic going in and out of copyleftistan. While this may not stop determined, technically-adept people, it will work for the most part. As I said, this sort of thing only really works on the margins.

Re: AI scrapers request commented scripts

#68

Post by kelnos »

nkrisc wrote: post #35
hsbauauvhabzb wrote: post #24

Earlier quoted context omitted.

How else do you tell the bot you do not wish to be scraped? Your analogy is lacking - you didn’t order a package, you never wanted a package, and the postman is taking something, not leaving it, and you’ve explicitly left a sign saying ‘you are not welcome here’.

Put your content behind authentication if you don’t want it to be requested by just anyone.

But I do want my content accessible to "just anyone", as long as they are humans. I don't want it accessible to bots.

You are free to say "well, there is no mechanism to do that", and I would agree with you. That's the problem!

Post reply on HN