Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI scrapers request commented scripts

cryptography.dog

Post reply on HN
Print view View original

221–230 of 234 posts

Re: AI scrapers request commented scripts

#221

Post by gkbrk »

grayhatter wrote: post #201
wqaatwt wrote: post #197

Earlier quoted context omitted.

IIRC Firefox (and I assume other browsers) when using privacy/no tracking mode does send fake data..

You're incorrect. I've never seen any browser, on it's own lie about it's user agent. (I can set a custom string and lie with it, but that's not the agent doing it) Do you have a specific / concrete example in mind? Or are you mistaking a feature from something other than a mainstream browser?

Firefox sends an incorrect version and operating system on its User-Agent when the privacy settings are turned on.

IIRC it defaults to a Windows user agent even when you use it on other operating systems.

Re: AI scrapers request commented scripts

#222

Post by gkbrk »

j2kun wrote: post #86
bakql wrote: post #20

>These were scrapers, and they were most likely trying to non-consensually collect content for training LLMs. "Non-consensually", as if you had to ask for permission to perform a GET request to an open HTTP server. Yes, I know about weev. That was a travesty.

You should not have to ask for permission, but you should have to honestly set your user-agent. (In my opinion, this should be the law and it should be enforced)

> In my opinion, this should be the law and it should be enforced

You think people should go to prison if they go to their browser settings and change their user agent?

Re: AI scrapers request commented scripts

#223

Post by aendruk »

gkbrk wrote: post #222
j2kun wrote: post #86

Earlier quoted context omitted.

You should not have to ask for permission, but you should have to honestly set your user-agent. (In my opinion, this should be the law and it should be enforced)

> In my opinion, this should be the law and it should be enforced You think people should go to prison if they go to their browser settings and change their user agent?

Corporation, fined

Re: AI scrapers request commented scripts

#224

Post by grayhatter »

gkbrk wrote: post #221
grayhatter wrote: post #201

Earlier quoted context omitted.

You're incorrect. I've never seen any browser, on it's own lie about it's user agent. (I can set a custom string and lie with it, but that's not the agent doing it) Do you have a specific / concrete example in mind? Or are you mistaking a feature from something other than a mainstream browser?

Firefox sends an incorrect version and operating system on its User-Agent when the privacy settings are turned on. IIRC it defaults to a Windows user agent even when you use it on other operating systems.

You're incorrect. I have Firefox configured with the most strict privacy settings, and it returns `Mozilla/5.0 (X11; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0)` With the exception of it being Wayland instead of X11 it's entirely accurate. Would love to see whatever gaslit you of something so easy to test and validate.

Re: AI scrapers request commented scripts

#225

Post by gkbrk »

grayhatter wrote: post #224
gkbrk wrote: post #221

Earlier quoted context omitted.

Firefox sends an incorrect version and operating system on its User-Agent when the privacy settings are turned on. IIRC it defaults to a Windows user agent even when you use it on other operating systems.

You're incorrect. I have Firefox configured with the most strict privacy settings, and it returns `Mozilla/5.0 (X11; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0)` With the exception of it being Wayland instead of X11 it's entirely accurate. Would love to see whatever gaslit you of something so easy to test and validate.

Nope, as it turns out this was actually a thing until 2025-01-24, where a commit removed this "pretend to be Windows even on Linux platforms" behavior.

https://github.com/mozilla-firefox/firefox/commit/eb2f90f870...

This OS spoofing behavior was added in 2019-01-09 with this commit:

https://github.com/mozilla-firefox/firefox/commit/264fe08c09...

So Firefox has spoofed the User-Agent as a Windows machine on Linux for around 6 years, and only stopped doing it early this year. Would love to see whatever gaslit you into forgetting this easy to test and validate behavior.

Re: AI scrapers request commented scripts

#226

Post by grayhatter »

gkbrk wrote: post #225
grayhatter wrote: post #224

Earlier quoted context omitted.

You're incorrect. I have Firefox configured with the most strict privacy settings, and it returns `Mozilla/5.0 (X11; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0)` With the exception of it being Wayland instead of X11 it's entirely accurate. Would love to see whatever gaslit you of something so easy to test and validate.

Nope, as it turns out this was actually a thing until 2025-01-24, where a commit removed this "pretend to be Windows even on Linux platforms" behavior. https://github.com/mozilla-firefox/firefox/commit/eb2f90f870... This OS spoofing behavior was added in 2019-01-09 with this commit: https://github.com/mozilla-firefox/firefox/commit/264fe08c09... So Firefox has spoofed the User-Agent as a Windows machine on Linux for…

https://support.mozilla.org/en-US/kb/resist-fingerprinting

This was part of the resist fingerprinting feature. Which is an advanced user configuration. I can alter the user agent directly myself too.

Sigh

I regret getting tricked into arguing over such a pedantic specific, So I'd like to redirect the actual point, which is that it's not meaningful if a Firefox browser pretends to be a slightly different Firefox browser, but instead the problem is when something that's not a browser, claims to be and behave like one.

Still, +1 for finding the commit, I'd forgotten about this feature. I thought only the tor browser was this foolish.

Re: AI scrapers request commented scripts

#227

Post by ang_cire »

ordu wrote: post #173
ang_cire wrote: post #84

Earlier quoted context omitted.

Yep. Robots.txt is a framework intended for performance, not a legal or ethical imperative. If you want to control how someone accesses something, the onus is on you to put access controls in place. The people who put things on a public, un-restricted server and then complain that the public accessed it in an un-restricted way might be excusable if it's some geocities-esque Mom and Pop site that has no reason to know…

It is an asocial state of mind. We have locks and security systems that prevent people from stealing. But if all people agreed to not steal, then we could save that efforts for something better. The ideal approach doesn't work with the stealing, and now it doesn't work with HTTP either. It just raises costs for a society with no lasting benefit for anyone: site owners just figure out ways to restrict access and no mo…

Visiting websites isn't theft. Scraping isn't theft. Copyright infringement isn't theft.

Turning a digital asset like web pages that have no intrinsic limits on how many times they can be accessed into a limited good because you don't want someone to have something is the actual asocial mindset.

Re: AI scrapers request commented scripts

#228

Post by smsm42 »

davsti4 wrote: post #32
hsbauauvhabzb wrote: post #24

Earlier quoted context omitted.

How else do you tell the bot you do not wish to be scraped? Your analogy is lacking - you didn’t order a package, you never wanted a package, and the postman is taking something, not leaving it, and you’ve explicitly left a sign saying ‘you are not welcome here’.

Its simple, and I'll quote myself - "robots.txt isn't the law".

"I will do anything that will not literally get me into jail" is a pretty low bar. Most decent people try to do better than that - and that's the only reason society still exists, because there's not enough cops to put all bad people into jail and never will be.

Re: AI scrapers request commented scripts

#229

Post by davsti4 »

bigbuppo wrote: post #213
davsti4 wrote: post #32

Earlier quoted context omitted.

Its simple, and I'll quote myself - "robots.txt isn't the law".

Violating norms makes you an abusive jerk at best.

That's a slippery slope opinion.

One person's norm can oppose another's norm. Who defines them? If my "norm" is to stop at a stop sign, and yours is to roll through it, are you the abusive jerk?

Re: AI scrapers request commented scripts

#230

Post by davsti4 »

smsm42 wrote: post #228
davsti4 wrote: post #32

Earlier quoted context omitted.

Its simple, and I'll quote myself - "robots.txt isn't the law".

"I will do anything that will not literally get me into jail" is a pretty low bar. Most decent people try to do better than that - and that's the only reason society still exists, because there's not enough cops to put all bad people into jail and never will be.

Exactly - its why laws are written. So, if its enough of a problem, then there should be a law stopping/regulating it.

Are published TOSs enough to stop the owners of the bots?

Post reply on HN