Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI scrapers request commented scripts

cryptography.dog

Post reply on HN
Print view View original

171–180 of 234 posts

Re: AI scrapers request commented scripts

#171

Post by anon10484810573 »

codyb wrote: post #56
bakql wrote: post #20

>These were scrapers, and they were most likely trying to non-consensually collect content for training LLMs. "Non-consensually", as if you had to ask for permission to perform a GET request to an open HTTP server. Yes, I know about weev. That was a travesty.

The sign on the door said "no scrapers", which as far as I know is not a protected class.

This mindset really baffles me. Just because it is not illegal doesn't mean one should do it. And for anything truly innovative there are bound to be gaps in the current law.

It's pretty obvious that there is an asymmetry in benefit between those creating the models and those creating the content. If that doesn't bother you consider the fact that this currently undermines the economic and social model for open content creation on the internet.

What happens when the content significantly decreases?

Should those who create content not have some say in how their content is used?

Re: AI scrapers request commented scripts

#172

Post by iamacyborg »

Mars008 wrote: post #158
overfeed wrote: post #156

Earlier quoted context omitted.

> Looks like it's time for in-browser scrappers. If scrapers were as well-behaved as humans, website operators wouldn't bother to block them[1]. It's the abuse that motivates the animus and action. As the fine articles spelt out, scrapers are greedy in many ways, one of which is trying to slurp down as many URLs as possible without wasting bytes. Not enough people know about common crawl, or know how to write multith…

The question is who runs them? There are only a few big companies like MS, Google, OpenAI, Anthropic. But from the posts here it looks like hordes of buggy scrapers run by enthusiasts.

Lots of “data” companies out there that want to sell you scraped data sets.

Re: AI scrapers request commented scripts

#173

Post by ordu »

ang_cire wrote: post #84
hexage1814 wrote: post #75

[flagged]

Yep. Robots.txt is a framework intended for performance, not a legal or ethical imperative. If you want to control how someone accesses something, the onus is on you to put access controls in place. The people who put things on a public, un-restricted server and then complain that the public accessed it in an un-restricted way might be excusable if it's some geocities-esque Mom and Pop site that has no reason to know…

It is an asocial state of mind. We have locks and security systems that prevent people from stealing. But if all people agreed to not steal, then we could save that efforts for something better. The ideal approach doesn't work with the stealing, and now it doesn't work with HTTP either. It just raises costs for a society with no lasting benefit for anyone: site owners just figure out ways to restrict access and no more scraping of pages that they do not want to be scraped.

A healthy society relies on a cooperation between members. It relies on them accepting some rules that limits their behavior. Like we agreed not to kill others, and now I can go outside without weapons and anti-bullet defenses.

Re: AI scrapers request commented scripts

#174

Post by mrweasel »

OhMeadhbh wrote: post #14

I blame modern CS programs that don't teach kids about parsing. The last time I looked at some scraping code, the dev was using regexes to "parse" html to find various references. Maybe that's a way to defend against bots that ignore robots.txt, include a reference to a Honeypot HTML file with garbage text, but include the link to it in a comment.

You don't need to teach parsing, that won't help much any way. We need to teach people to be good netizen again. I'd argue that it was always viewed as reasonable to scrape content, as long as you didn't misrepresent content as your own and if you scraped responsibly, backing of if the server started to slow down, or simply not crawling to fast to begin with.

Currently we have at least three problems:

1) Companies have no issue with not providing sources and not linking back.

2) There are too many scrapers, even if they behaved, some site would struggle to handle all of them.

3) Srapers go full throttle 24/7, expecting the sites to rate-limit them if they are going to fast. Hammer a site into the ground, just wait until it's back and hammer it again, grabbing what you can before it crashes once more.

There's no longer a sense of the internet being for all of us and that we need to make room for each other. Website / human generated content exists as a resource to be strip mined.

Re: AI scrapers request commented scripts

#176

Post by lucasluitjes »

throw_me_uwu wrote: post #175

> most likely trying to non-consensually collect content for training LLMs No, it's just background internet scanning noise

This.

If you were writing a script to mass-scan the web for vulnerabilities, you would want to collect as many http endpoints as possible. JS files, regardless of whether they're commented out or not, are a great way to find endpoints in modern web applications.

If you were writing a scraper to collect source code to train LLMs on, I doubt you would care as much about a commented-out JS file. I'm not sure you'd even want to train on random low-quality JS served by websites. Anyone familiar with LLM training data collection who can comment on this?

Re: AI scrapers request commented scripts

#177

Post by distances »

halJordan wrote: post #39
yuliyp wrote: post #36

Earlier quoted context omitted.

Having a front door physically allows anyone on the street to come to knock on it. Having a "no soliciting" sign is an instruction clarifying that not everybody is welcome. Having a web site should operate in a similar fashion. The robots.txt is the equivalent of such a sign.

No soliciting signs are polite requests that no one has to follow, and door to door salesman regularly walk right past them. No one is calling for the criminalization of door-to-door sales and no one is worried about how much door-to-door sales increases water consumption.

> No one is calling for the criminalization of door-to-door sales

Door-to-door sales absolutely are banned in many jurisdictions.

Re: AI scrapers request commented scripts

#179

Post by meepmorp »

dylan604 wrote: post #114
sdenton4 wrote: post #45

Earlier quoted context omitted.

The problem is that serving content costs money. Llm scraping is essentially ddos'ing content meant for human consumption. Ddos'ing sucks.

running the scraping bots cost money too.

> Won’t somebody please think of the parasites?

Re: AI scrapers request commented scripts

#180

Post by herbst »

2OEH8eoCRo0 wrote: post #46
sdenton4 wrote: post #45

Earlier quoted context omitted.

The problem is that serving content costs money. Llm scraping is essentially ddos'ing content meant for human consumption. Ddos'ing sucks.

Scraping is legal. DDoSing isn't. We should start suing these bad actors. Why do techies forget that the legal system exists?

Facebook and Bing sometimes are 80% of my daily hits and don't respect my IP bans and other bot filterings at all. You think I can just sue them and have any change to win before being broke?
Post reply on HN