Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI scrapers request commented scripts

cryptography.dog

Post reply on HN
Print view View original

201–210 of 234 posts

Re: AI scrapers request commented scripts

#201

Post by grayhatter »

wqaatwt wrote: post #197
grayhatter wrote: post #196

Earlier quoted context omitted.

Someone using a VPN is not lying. The intent of a user agent is to identify the software sending the request. The IP address isn't sent by the browser, and isn't part of the HTTP request. It's part of the routing information required to deliver the packet back to the client. If a client sent it's "real" IP address as an HTTP header, and I tried to respond to that IP instead of the IP address from the TCP packet. It w…

IIRC Firefox (and I assume other browsers) when using privacy/no tracking mode does send fake data..

You're incorrect. I've never seen any browser, on it's own lie about it's user agent. (I can set a custom string and lie with it, but that's not the agent doing it)

Do you have a specific / concrete example in mind? Or are you mistaking a feature from something other than a mainstream browser?

Re: AI scrapers request commented scripts

#202

Post by batch12 »

grayhatter wrote: post #198
gkbrk wrote: post #191

Earlier quoted context omitted.

A server returning HTML for Chrome but not cURL seems like a bug, no? This is why there are so many libraries to make requests that look like they came from browser, to work around buggy servers or server operators with wrong assumptions.

> A server returning HTML for Chrome but not cURL seems like a bug, no? tell me you've never heard of https://wttr.in/ without telling me. :P It would absolutely be a bug iff this site returned html to curl. > This is why there are so many libraries to make requests that look like they came from browser, to work around buggy servers or server operators with wrong assumptions. This is a shallow take, the best counter…

Googlebot has no problem identifying itself because Google knows that you want it to index your site if you want visitors. It doesn't identify itself to give you the option to block it. It identifies itself so you don't.

Re: AI scrapers request commented scripts

#203

Post by grayhatter »

batch12 wrote: post #199
jraph wrote: post #165

Earlier quoted context omitted.

Lies in user agent strings where for bypassing bugs, poor workarounds and assumptions that became wrong, they are nothing like what we are talking about.

Yes, the client wanted the server to deliver content it had intended for a different client, regardless of what the service operator wanted, so it lied using its user agent. Exact same thing we are talking about. The difference is that people don't want companies to profit off of their content. That's fair. In this case, they should maybe consider some form of real authentication, or if the bot is abusive, some kind…

> Yes, the client wanted the server to deliver content it had intended for a different client, regardless of what the service operator wanted, so it lied using its user agent.

I would actually argue, it's not nearly the same type of misconfiguration. The reason scripts, which have never been a browser, who omit their real identity, are doing it, is to evade bot detection. The reason browsers pack their UA with so much legacy data, is because of misconfigured servers. The server owner wants to send data to users and their browsers, but through incompetence, they've made a mistake. Browsers adapted by including extra strings in the UA to account for the expectations of incorrectly configured servers. Extra strings being the critical part, Google bot's UA is an example of this being done correctly.

Re: AI scrapers request commented scripts

#204

Post by jMyles »

LexGray wrote: post #96
jMyles wrote: post #50

Earlier quoted context omitted.

Sexual consent is sacred. This metaphor is in truly bad taste. When you return a response with a 200-series status code, you've granted consent. If you don't want to grant consent, change the logic of the server.

Perhaps bad taste, but bots could also be legitimately purposely violating the most private or traumatizing moments a vulnerable person has in any exploitative way it cares to. I am not sure using bad taste is enough of an excuse to not discuss the issue as many people do in fact use the internet for sexual things. If anything consent should be MORE important because it is easier to document and verify. A vast hoard…

> the most private or traumatizing moments a vulnerable person has

...and in this hypothetical, this person is serving them via an unauthenticated http server and hoping that clients will respect robots.txt?

Re: AI scrapers request commented scripts

#205

Post by jMyles »

dns_snek wrote: post #182
jMyles wrote: post #116

Earlier quoted context omitted.

I guess I'm more optimistic about the future of the human condition. > You use "legacy" as if these systems are obsolete and on their way out. They're not. I have serious doubts that nation states will still exist in 500 years. I feel quite certain that they'll be gone in 10,000. And I think it's generally good to build an internet for those time scales. > base arguments on your preferred vision of how things should…

> Again, maybe I'm just more optimistic, but I think that on longer time frames, the sober elder statesmen/women will prevail and realize that violence is not an appropriate response to bytes transiting the wire that they wish weren't. Your framing is off because this notion of fairness or morality isn't something they concern themselves with. They're using violence because if they didn't, they would be allowing othe…

I mean... I agree with everything you've said here. I'm not sure what makes you think I've mis-framed the stakes.

Re: AI scrapers request commented scripts

#206

Post by grayhatter »

batch12 wrote: post #202
grayhatter wrote: post #198

Earlier quoted context omitted.

> A server returning HTML for Chrome but not cURL seems like a bug, no? tell me you've never heard of https://wttr.in/ without telling me. :P It would absolutely be a bug iff this site returned html to curl. > This is why there are so many libraries to make requests that look like they came from browser, to work around buggy servers or server operators with wrong assumptions. This is a shallow take, the best counter…

Googlebot has no problem identifying itself because Google knows that you want it to index your site if you want visitors. It doesn't identify itself to give you the option to block it. It identifies itself so you don't.

I care much less about being indexed by Google as much as you might think.

Google bot doesn't get blocked from my server primarily because it's a *very* well behaved bot. It sends a lot of requests, but it's very kind, and has never acted in a way that could overload my server. It respects robots.txt, and identifies itself multiple times.

Google bot doesn't get blocked, because it's a well behaved bot that eagerly follows the rules. I wouldn't underestimate how far that goes towards the reason it doesn't get blocked. Much more than the power gained by being google search.

Re: AI scrapers request commented scripts

#207

Post by bigbuppo »

jMyles wrote: post #204
LexGray wrote: post #96

Earlier quoted context omitted.

Perhaps bad taste, but bots could also be legitimately purposely violating the most private or traumatizing moments a vulnerable person has in any exploitative way it cares to. I am not sure using bad taste is enough of an excuse to not discuss the issue as many people do in fact use the internet for sexual things. If anything consent should be MORE important because it is easier to document and verify. A vast hoard…

> the most private or traumatizing moments a vulnerable person has ...and in this hypothetical, this person is serving them via an unauthenticated http server and hoping that clients will respect robots.txt?

Robots are supposed to behave. It was a solved problem 30 years ago until AI bros unsolved it. Any entity that does not obey robots.txt is by definition a malicious actor.

Re: AI scrapers request commented scripts

#208

Post by bigbuppo »

Larrikin wrote: post #60
jMyles wrote: post #50

Earlier quoted context omitted.

Sexual consent is sacred. This metaphor is in truly bad taste. When you return a response with a 200-series status code, you've granted consent. If you don't want to grant consent, change the logic of the server.

[flagged]

They also conveniently missed the point that it was about victim blaming.

Re: AI scrapers request commented scripts

#209

Post by ToucanLoucan »

kijin wrote: post #153
Aloisius wrote: post #137

Earlier quoted context omitted.

Legitimate software that scan links are often well behaved, in isolation. It's when that software is installed on millions of computers that in aggregate, they can behave poorly. This isn't particularly new though. RSS software used to blow up small websites that couldn't handle it. Now with some browsers speculatively loading links, you can be hammered simply because you're linked to from a popular site even if no o…

Yes, and sysadmins have been quietly banning those misbehaving programs for the last 30 years. The only thing that seems to have changed is that today's thread is full of people who think they have some sort of human right to access any website by any means possible, including their sloppy vibe-coded crawler. In the past, IIRC, people used to be a little more apologetic about consuming other people's resources and di…

The entitlement of so many modern vibe coders (or as we called them before, script kiddies) is absolutely off the charts. Just because there is not a rule or law expressly against what you're doing doesn't mean it's perfectly fine to do. Websites are hosted by and funded by people, and if your shitty scraper racks up a ton of traffic on one of my sites, I may end up on the hook for that. I am perfectly within both my rights and ethical boundaries to block your IP(s).

And just to not leave it merely implied, I don't give a rats ass if that slows down your "innovation." Go away.

Re: AI scrapers request commented scripts

#210

Post by jraph »

batch12 wrote: post #199
jraph wrote: post #165

Earlier quoted context omitted.

Lies in user agent strings where for bypassing bugs, poor workarounds and assumptions that became wrong, they are nothing like what we are talking about.

Yes, the client wanted the server to deliver content it had intended for a different client, regardless of what the service operator wanted, so it lied using its user agent. Exact same thing we are talking about. The difference is that people don't want companies to profit off of their content. That's fair. In this case, they should maybe consider some form of real authentication, or if the bot is abusive, some kind…

Add "assumptions that became wrong" to "intended" and the perspective radically changes, to the point that omitting this part from my comment changes everything.

I would even add:

> the client wanted the server to deliver content it had intended for a different client

In most cases, the webmaster intended their work to look good, not really to send different content to different clients. That later part is a technical means, a workaround. The intent of bringing the ok version to the end user was respected… even better with the user agent lies!

> The difference is that people don't want companies to profit off of their content.

Indeed¹, and also they don't want terrible bot to bring down their servers.

1: well, my open source work explicitly allows people to profit off of it - as long as the license is respected (attribution, copyleft, etc)

Post reply on HN