Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI scrapers request commented scripts

cryptography.dog

Post reply on HN
Print view View original

191–200 of 234 posts

Re: AI scrapers request commented scripts

#191

Post by gkbrk »

jraph wrote: post #165
batch12 wrote: post #136

Earlier quoted context omitted.

Browser user agents have a history of being lies from the earliest days of usage. Official browsers lied about what they were- and still do.

Lies in user agent strings where for bypassing bugs, poor workarounds and assumptions that became wrong, they are nothing like what we are talking about.

A server returning HTML for Chrome but not cURL seems like a bug, no?

This is why there are so many libraries to make requests that look like they came from browser, to work around buggy servers or server operators with wrong assumptions.

Re: AI scrapers request commented scripts

#193

Post by 1vuio0pswjnm7 »

renegat0x0 wrote: post #99

Most web scrapers, even if illegal, are for... business. So they scrape amazon, or shops. So yeah. Most unwanted traffic is from big tech, or bad actors trying to sniff vulnerabilities. I know a thing or two about web scraping. There are sometimes status codes 404 for protection, so that you skip this site, so my crawler tries, as a hammer, several of faster crawling methods (curlcffi). Zip bombs are also not for me.…

Is there a difference between "scraping" and "crawling"

Re: AI scrapers request commented scripts

#194

Post by nkrisc »

kelnos wrote: post #68
nkrisc wrote: post #35

Earlier quoted context omitted.

Put your content behind authentication if you don’t want it to be requested by just anyone.

But I do want my content accessible to "just anyone", as long as they are humans. I don't want it accessible to bots. You are free to say "well, there is no mechanism to do that", and I would agree with you. That's the problem!

Even abusive crawlers and scrapers are acting as agents of real humans, just as your browser is acting as your agent. I don't even know how you could reliably draw a reasonable line in the sand between the two without putting some group of people on the wrong side of the line.

I suppose the ultimate solution would be browsers and operating systems and hardware manufacturers co-operating to implement some system that somehow cryptographically signs HTTP requests which attests that it was triggered by an actual, physical interaction with a computing device by a human.

Though you don't have to think for very long to come up with all kinds of collateral damage that would cause and how bad actors could circumvent it anyway.

All in all, this whole issue seems more like a legal problem than a technical one.

Re: AI scrapers request commented scripts

#195

Post by dragonwriter »

hsbauauvhabzb wrote: post #154
dragonwriter wrote: post #146

Earlier quoted context omitted.

Generally, legally, no, not just for ignoring a “no soliciting” sign.

But they’re presumably trespassing.

And, despite what ideas you may get from the media, mere trespass without imminent threat to life is not a justification for deadly force.

There are some states where the considerations for self defense do not include a duty to retreat if possible, either in general (“stand your ground" law) or specifically in the home (“castle doctrine"), but all the other requirements (imminent threat of certain kinds of serious harm, proportional force) for self-defense remain part of the law in those states, and trespassing by/while disregarding a ”no soliciting” would not, by itself, satisfy those requirements.

Re: AI scrapers request commented scripts

#196

Post by grayhatter »

wqaatwt wrote: post #187
grayhatter wrote: post #106

Earlier quoted context omitted.

If you're lying in the requests you send, to trick my server into returning the content you want, instead of what I would want to return to webscrapers, that's non-consensual. You don't need my permission to send a GET request, I completely agree. In fact, by having a publicly accessible webserver, there's implied consent that I'm willing to accept reasonable, and valid GET requests. But I have configured my server t…

Somebody concealing or obfuscating various information a browser would send by standard for privacy or other reasons is also “lying” by that standard? Or someone using a VPN?

Someone using a VPN is not lying. The intent of a user agent is to identify the software sending the request. The IP address isn't sent by the browser, and isn't part of the HTTP request. It's part of the routing information required to deliver the packet back to the client. If a client sent it's "real" IP address as an HTTP header, and I tried to respond to that IP instead of the IP address from the TCP packet. It would never arrive.

There's a difference between sending no data, and sending false data. I don't block requests without http referrers for that very reason.

Re: AI scrapers request commented scripts

#197

Post by wqaatwt »

grayhatter wrote: post #196
wqaatwt wrote: post #187

Earlier quoted context omitted.

Somebody concealing or obfuscating various information a browser would send by standard for privacy or other reasons is also “lying” by that standard? Or someone using a VPN?

Someone using a VPN is not lying. The intent of a user agent is to identify the software sending the request. The IP address isn't sent by the browser, and isn't part of the HTTP request. It's part of the routing information required to deliver the packet back to the client. If a client sent it's "real" IP address as an HTTP header, and I tried to respond to that IP instead of the IP address from the TCP packet. It w…

IIRC Firefox (and I assume other browsers) when using privacy/no tracking mode does send fake data..

Re: AI scrapers request commented scripts

#198

Post by grayhatter »

gkbrk wrote: post #191
jraph wrote: post #165

Earlier quoted context omitted.

Lies in user agent strings where for bypassing bugs, poor workarounds and assumptions that became wrong, they are nothing like what we are talking about.

A server returning HTML for Chrome but not cURL seems like a bug, no? This is why there are so many libraries to make requests that look like they came from browser, to work around buggy servers or server operators with wrong assumptions.

> A server returning HTML for Chrome but not cURL seems like a bug, no?

tell me you've never heard of https://wttr.in/ without telling me. :P

It would absolutely be a bug iff this site returned html to curl.

> This is why there are so many libraries to make requests that look like they came from browser, to work around buggy servers or server operators with wrong assumptions.

This is a shallow take, the best counter example is how googlebot has no problem identifying it itself both in and out of thue user agent. Do note user agent packing, is distinctly different from a fake user agent selected randomly from the list of most common.

The existence of many libraries with the intent to help conceal the truth about a request doesn't feel like proof that's what everyone should be doing. It feels more like proof that most people only want to serve traffic to browsers and real users. And it's the bots and scripts that are the fuckups.

Re: AI scrapers request commented scripts

#199

Post by batch12 »

jraph wrote: post #165
batch12 wrote: post #136

Earlier quoted context omitted.

Browser user agents have a history of being lies from the earliest days of usage. Official browsers lied about what they were- and still do.

Lies in user agent strings where for bypassing bugs, poor workarounds and assumptions that became wrong, they are nothing like what we are talking about.

Yes, the client wanted the server to deliver content it had intended for a different client, regardless of what the service operator wanted, so it lied using its user agent. Exact same thing we are talking about. The difference is that people don't want companies to profit off of their content. That's fair. In this case, they should maybe consider some form of real authentication, or if the bot is abusive, some kind of rate limiting control.

Re: AI scrapers request commented scripts

#200

Post by grayhatter »

batch12 wrote: post #136
grayhatter wrote: post #106

Earlier quoted context omitted.

If you're lying in the requests you send, to trick my server into returning the content you want, instead of what I would want to return to webscrapers, that's non-consensual. You don't need my permission to send a GET request, I completely agree. In fact, by having a publicly accessible webserver, there's implied consent that I'm willing to accept reasonable, and valid GET requests. But I have configured my server t…

Browser user agents have a history of being lies from the earliest days of usage. Official browsers lied about what they were- and still do.

Can you give a single example of a browser with a user agent that lies about it's real origin?

The best I can come up with is the TOR browser, which will reduce the number of bits of information it will return, but I dont consider that to be misleading. It's a custom build of firefox, that discloses it is firefox, and otherwise behaves exactly as I would expect firefox to behave.

Post reply on HN