Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI scrapers request commented scripts

cryptography.dog

Post reply on HN
Print view View original

51–60 of 234 posts

Re: AI scrapers request commented scripts

#51

Post by jraph »

bakql wrote: post #20

>These were scrapers, and they were most likely trying to non-consensually collect content for training LLMs. "Non-consensually", as if you had to ask for permission to perform a GET request to an open HTTP server. Yes, I know about weev. That was a travesty.

When I open an HTTP server to the public web, I expect and welcome GET requests in general.

However,

(1) there's a difference between (a) a regular user browsing my websites and (b) robots DDoSing them. It was never okay to hammer a webserver. This is not new, and it's for this reason that curl has had options to throttle repeated requests to servers forever. In real life, there are many instances of things being offered for free, it's usually not okay to take it all. Yes, this would be abuse. And no, the correct answer to such a situation would not be "but it was free, don't offer it for free if you don't want it to be taken for free". Same thing here.

(2) there's a difference between (a) a regular user reading my website or even copying and redistributing my content as long as the license of this work / the fair use or related laws are respected, and (b) a robot counterfeiting it (yeah, I agree with another commenter, theft is not the right word, let's call a spade a spade)

(3) well-behaved robots are expected to respect robots.txt. This is not the law, this is about being respectful. It is only fair bad-behaved robots get called out.

Well behaved robots do not usually use millions of residential IPs through shady apps to "Perform a get request to an open HTTP server".

Re: AI scrapers request commented scripts

#53

Post by [deleted] »

bakql wrote: post #20

>These were scrapers, and they were most likely trying to non-consensually collect content for training LLMs. "Non-consensually", as if you had to ask for permission to perform a GET request to an open HTTP server. Yes, I know about weev. That was a travesty.

[deleted]

Re: AI scrapers request commented scripts

#54

Post by andoando »

Calavar wrote: post #27
hsbauauvhabzb wrote: post #24

Earlier quoted context omitted.

How else do you tell the bot you do not wish to be scraped? Your analogy is lacking - you didn’t order a package, you never wanted a package, and the postman is taking something, not leaving it, and you’ve explicitly left a sign saying ‘you are not welcome here’.

If you are serving web pages, you are soliciting GET requests, kind of like ordering a package is soliciting a delivery. "Taking" versus "giving" is neither here nor there for this discussion. The question is are you expressing a preference on etiquette versus a hard rule that must be followed. I personally believe robots.txt is the former, and I say that as someone who serves more pages than they scrape

Well yes this is exactly what's happening as of now. But there SHOULD be a way to upload content without giving it access to scrapers.

Re: AI scrapers request commented scripts

#55

Post by jMyles »

munk-a wrote: post #41
Calavar wrote: post #27

Earlier quoted context omitted.

If you are serving web pages, you are soliciting GET requests, kind of like ordering a package is soliciting a delivery. "Taking" versus "giving" is neither here nor there for this discussion. The question is are you expressing a preference on etiquette versus a hard rule that must be followed. I personally believe robots.txt is the former, and I say that as someone who serves more pages than they scrape

I disagree strongly here - though not from a technical perspective. There's absolutely a legal concept of making your work available for viewing without making it available for copying and AI scraping (while we can technically phrase it as just viewing a bunch of times) is effectively copying. Lets say a large art hosting site realizes how damaging AI training on their data can be - should they respond by adding a pa…

> There's absolutely a legal concept of making your work available for viewing without making it available for copying

This "legal concept" is enforceable through legacy systems of police and violence. The internet does not recognize it. How much more obvious can this get?

If we stumble down the path of attempting to apply this legal framework, won't some jurisdiction arise with no IP protections whatsoever and just come to completely dominate the entire economy of the internet?

If I can spin up a server in copyleftistan with a complete copy of every album and film ever made, available for free download, why would users in copyrightistan use the locked down services of their domestic economy?

Re: AI scrapers request commented scripts

#56

Post by codyb »

bakql wrote: post #20

>These were scrapers, and they were most likely trying to non-consensually collect content for training LLMs. "Non-consensually", as if you had to ask for permission to perform a GET request to an open HTTP server. Yes, I know about weev. That was a travesty.

The sign on the door said "no scrapers", which as far as I know is not a protected class.

Re: AI scrapers request commented scripts

#57

Post by ColinWright »

2OEH8eoCRo0 wrote: post #46
sdenton4 wrote: post #45

Earlier quoted context omitted.

The problem is that serving content costs money. Llm scraping is essentially ddos'ing content meant for human consumption. Ddos'ing sucks.

Scraping is legal. DDoSing isn't. We should start suing these bad actors. Why do techies forget that the legal system exists?

There is no way that you can sue the people responsible for DDoSing your system. Even if you can find them ... and you won't ... they're likely as not either not in your jurisdiction (they might be in Russia, or China, or Bolivia, or anywhere) and they will have a lot more money than you.

People here on HN are laughing at the UKs Online Safety Act for trying to impose restrictions on people in other countries, and yet now you're implying that similar restrictions can be placed on people in other countries and over whom you have neither power nor control.

Re: AI scrapers request commented scripts

#58

Post by jraph »

jMyles wrote: post #50
bigbuppo wrote: post #49

Earlier quoted context omitted.

Seriously. Did you see what that web server was wearing? I mean, sure it said "don't touch me" and started screaming for help and blocked 99.9% of our IP space, but we got more and they didn't block that so clearly they weren't serious. They were asking for it. It's their fault. They're not really victims.

Sexual consent is sacred. This metaphor is in truly bad taste. When you return a response with a 200-series status code, you've granted consent. If you don't want to grant consent, change the logic of the server.

> When you return a response with a 200-series status code, you've granted consent. If you don't want to grant consent, change the logic of the server.

"If you don't consent to me entering your house, change its logic so that picking the door's lock doesn't let me open the door"

Yeah, well…

As if the LLM scrappers didn't try anything under the sun like using millions of different residential IP to prevent admins from "changing the logic of the server" so it doesn't "return a response with a 200-series status code" when they don't agree to this scrapping.

As if there weren't broken assumptions that make "When you return a response with a 200-series status code, you've granted consent" very false.

As if technical details were good carriers of human intents.

Re: AI scrapers request commented scripts

#59

Post by czscout »

yuliyp wrote: post #36
Calavar wrote: post #27

Earlier quoted context omitted.

If you are serving web pages, you are soliciting GET requests, kind of like ordering a package is soliciting a delivery. "Taking" versus "giving" is neither here nor there for this discussion. The question is are you expressing a preference on etiquette versus a hard rule that must be followed. I personally believe robots.txt is the former, and I say that as someone who serves more pages than they scrape

Having a front door physically allows anyone on the street to come to knock on it. Having a "no soliciting" sign is an instruction clarifying that not everybody is welcome. Having a web site should operate in a similar fashion. The robots.txt is the equivalent of such a sign.

And a no soliciting sign is no more cosmically binding than robots.txt. It's a request, not an enforceable command.

Re: AI scrapers request commented scripts

#60

Post by Larrikin »

jMyles wrote: post #50
bigbuppo wrote: post #49

Earlier quoted context omitted.

Seriously. Did you see what that web server was wearing? I mean, sure it said "don't touch me" and started screaming for help and blocked 99.9% of our IP space, but we got more and they didn't block that so clearly they weren't serious. They were asking for it. It's their fault. They're not really victims.

Sexual consent is sacred. This metaphor is in truly bad taste. When you return a response with a 200-series status code, you've granted consent. If you don't want to grant consent, change the logic of the server.

[flagged]
Post reply on HN