Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI scrapers request commented scripts

cryptography.dog

Post reply on HN
Print view View original

161–170 of 234 posts

Re: AI scrapers request commented scripts

#161

Post by duskdozer »

halJordan wrote: post #39
yuliyp wrote: post #36

Earlier quoted context omitted.

Having a front door physically allows anyone on the street to come to knock on it. Having a "no soliciting" sign is an instruction clarifying that not everybody is welcome. Having a web site should operate in a similar fashion. The robots.txt is the equivalent of such a sign.

No soliciting signs are polite requests that no one has to follow, and door to door salesman regularly walk right past them. No one is calling for the criminalization of door-to-door sales and no one is worried about how much door-to-door sales increases water consumption.

>No one is calling for the criminalization of door-to-door sales

Ok, I am, right now.

It seems like there are two sides here that are talking past one another: "people will do X and you accept it if you do not actively prevent it, if you can" and "X is bad behavior that should be stopped and shouldn't be the burden of individuals to stop". As someone who leans to the latter, the former just sounds like restating the problem being complained about.

Re: AI scrapers request commented scripts

#162

Post by jraph »

jack_pp wrote: post #149
jraph wrote: post #90

Earlier quoted context omitted.

> Either require a fee (processing power, captcha etc) or make a private password (auth) Well, I shouldn't have to work or make things worse for everybody because the LLM bros decided to screw us. > It is inherently a cat and mouse game that you CHOOSE to play No, let's not reverse the roles and blame the victims here. We sysadmins and authors are willing to share our work publicly to the world but never asked for it…

That's like saying you shouldn't have to sanitize your database inputs because you never asked for people to SQL inject your database. This stance is truly mind boggling to me

Would you take the defense of attackers using SQL injections? Because it feels like people here, including you, are defending the llm scrapers against sysadmins and authors who dare share their work publicly.

Ensuring basic security and robustness of a piece of software is simply not remotely comparable to countering the abuse these llm companies carry on.

But it's not even the point. And preventing SQL injections (through healthy programming practices) doesn't make things worse for any legitimate user neither.

Re: AI scrapers request commented scripts

#164

Post by catlifeonmars »

jack_pp wrote: post #149
jraph wrote: post #90

Earlier quoted context omitted.

> Either require a fee (processing power, captcha etc) or make a private password (auth) Well, I shouldn't have to work or make things worse for everybody because the LLM bros decided to screw us. > It is inherently a cat and mouse game that you CHOOSE to play No, let's not reverse the roles and blame the victims here. We sysadmins and authors are willing to share our work publicly to the world but never asked for it…

That's like saying you shouldn't have to sanitize your database inputs because you never asked for people to SQL inject your database. This stance is truly mind boggling to me

It’s both. You should sanitize your inputs because there are bad actors, but you also categorize attempts to sql inject as abuse and there is legal recourse.

Re: AI scrapers request commented scripts

#165

Post by jraph »

batch12 wrote: post #136
grayhatter wrote: post #106

Earlier quoted context omitted.

If you're lying in the requests you send, to trick my server into returning the content you want, instead of what I would want to return to webscrapers, that's non-consensual. You don't need my permission to send a GET request, I completely agree. In fact, by having a publicly accessible webserver, there's implied consent that I'm willing to accept reasonable, and valid GET requests. But I have configured my server t…

Browser user agents have a history of being lies from the earliest days of usage. Official browsers lied about what they were- and still do.

Lies in user agent strings where for bypassing bugs, poor workarounds and assumptions that became wrong, they are nothing like what we are talking about.

Re: AI scrapers request commented scripts

#166

Post by eur0pa »

Mars008 wrote: post #141
renegat0x0 wrote: post #99

Most web scrapers, even if illegal, are for... business. So they scrape amazon, or shops. So yeah. Most unwanted traffic is from big tech, or bad actors trying to sniff vulnerabilities. I know a thing or two about web scraping. There are sometimes status codes 404 for protection, so that you skip this site, so my crawler tries, as a hammer, several of faster crawling methods (curlcffi). Zip bombs are also not for me.…

Looks like it's time for in-browser scrappers. They will be indistinguishable from the servers side. With AI driver can pass even human tests.

you mean OpenAI Atlas?

Re: AI scrapers request commented scripts

#167

Post by marginalia_nu »

rokkamokka wrote: post #2

I'm not overly surprised, it's probably faster to search the text for http/https than parse the DOM

The regex approach is certainly easier to implement, but honestly static DOM parsing is pretty cheap, but quite fiddly to get right. You're probably gonna be limited by network congestion (or ephemeral ports) before you run out of CPU time doing this type of crawling.

Re: AI scrapers request commented scripts

#168

Post by Y-bar »

klntsky wrote: post #160
Cervisia wrote: post #76

Earlier quoted context omitted.

> robots.txt. This is not the law In Germany, it is the law. § 44b UrhG says (translated): (1) Text and data mining is the automated analysis of one or more digital or digitized works to obtain information, in particular about patterns, trends, and correlations. (2) Reproductions of lawfully accessible works for text and data mining are permitted. These reproductions must be deleted when they are no longer needed for…

> A reservation of rights for works accessible online is only effective if it is in machine-readable form. What if MY machine can't read it though?

That’s your problem.

A solution has been offered and you can adhere to it, or stop doing that thing which causes problems for many of us.

Re: AI scrapers request commented scripts

#169

Post by anon10484810573 »

2OEH8eoCRo0 wrote: post #46
sdenton4 wrote: post #45

Earlier quoted context omitted.

The problem is that serving content costs money. Llm scraping is essentially ddos'ing content meant for human consumption. Ddos'ing sucks.

Scraping is legal. DDoSing isn't. We should start suing these bad actors. Why do techies forget that the legal system exists?

> Why do techies forget that the legal system exists?

Simple, the gamble is it often makes business sense to "forget". Initially it could be they are unaware of the specific law but after a certain period of time it can really only be assumed the motivation is it is more convenient to ignore it.

Not all techies are like this.

Until the regulatory system corrects this calculus it will keep happening. Reputational costs or social costs sure wont correct it in this day and age.

Re: AI scrapers request commented scripts

#170

Post by bartread »

Mars008 wrote: post #141
renegat0x0 wrote: post #99

Most web scrapers, even if illegal, are for... business. So they scrape amazon, or shops. So yeah. Most unwanted traffic is from big tech, or bad actors trying to sniff vulnerabilities. I know a thing or two about web scraping. There are sometimes status codes 404 for protection, so that you skip this site, so my crawler tries, as a hammer, several of faster crawling methods (curlcffi). Zip bombs are also not for me.…

Looks like it's time for in-browser scrappers. They will be indistinguishable from the servers side. With AI driver can pass even human tests.

Not a new idea. For years now, on the occasions I’ve needed to scrape, I’ve used a set of ViolentMonkey scripts. I’ve even considered creating an extension, but have never really needed it enough to do the extra work.

But this is why lots of sites implement captchas and other mechanisms to detect, frustrate, or trap automated activity - because plenty of bots run in browsers too.

Post reply on HN