[flagged]
[dead]
AI scrapers request commented scripts
181–190 of 234 posts
Re: AI scrapers request commented scripts
#182Earlier quoted context omitted.
> legacy systems of police and violence You use "legacy" as if these systems are obsolete and on their way out. They're not. They're here to stay, and will remain dominant, for better or worse. Calling them "legacy" feels a bit childish, as if you're trying to ignore reality and base arguments on your preferred vision of how things should be. > The internet does not recognize it. Sure it does. Not universally, but th…
I guess I'm more optimistic about the future of the human condition. > You use "legacy" as if these systems are obsolete and on their way out. They're not. I have serious doubts that nation states will still exist in 500 years. I feel quite certain that they'll be gone in 10,000. And I think it's generally good to build an internet for those time scales. > base arguments on your preferred vision of how things should…
> Again, maybe I'm just more optimistic, but I think that on longer time frames, the sober elder statesmen/women will prevail and realize that violence is not an appropriate response to bytes transiting the wire that they wish weren't.
Your framing is off because this notion of fairness or morality isn't something they concern themselves with. They're using violence because if they didn't, they would be allowing other entities to gain wealth and power at their expense. I don't think it's much more complex than that.
See how differently these same bytes are treated in the hands of Aaron Swartz vs OpenAI. One threatened to empower humanity at the expense of reducing profits for a few rich men, so he got crucified for it. The other is hoping to make humans redundant, concentrate the distribution of wealth even further, and strengthen the US world dominance, so all of the right wheels get greased for them and they get a license to kill - figuratively and literally.
Re: AI scrapers request commented scripts
#183Earlier quoted context omitted.
Seriously. Did you see what that web server was wearing? I mean, sure it said "don't touch me" and started screaming for help and blocked 99.9% of our IP space, but we got more and they didn't block that so clearly they weren't serious. They were asking for it. It's their fault. They're not really victims.
Sexual consent is sacred. This metaphor is in truly bad taste. When you return a response with a 200-series status code, you've granted consent. If you don't want to grant consent, change the logic of the server.
Future rapist right here.
Re: AI scrapers request commented scripts
#184Most web scrapers, even if illegal, are for... business. So they scrape amazon, or shops. So yeah. Most unwanted traffic is from big tech, or bad actors trying to sniff vulnerabilities. I know a thing or two about web scraping. There are sometimes status codes 404 for protection, so that you skip this site, so my crawler tries, as a hammer, several of faster crawling methods (curlcffi). Zip bombs are also not for me.…
content-length is computed after content-encoding
If it’s present at all.
Re: AI scrapers request commented scripts
#185Earlier quoted context omitted.
But I do want my content accessible to "just anyone", as long as they are humans. I don't want it accessible to bots. You are free to say "well, there is no mechanism to do that", and I would agree with you. That's the problem!
What the hell? That is incredibly discriminatory. Fuck off. I support those that counter those discriminatory mechanisms.
Discriminatory against bots? That doesn't even make any sense.
Re: AI scrapers request commented scripts
#186Two thoughts here when it comes to poisoning unwanted LLM training data traffic 1) A coordinated effort among different sites will have a much greater chance of poisoning the data of a model so long as they can avoid any post scraping deduplication or filtering. 2) I wonder if copyright law can be used to amplify the cost of poisoning here. Perhaps if the poisoned content is something which has already been shown to…
As for 1, it would be great to have this as a plugin for WordPress etc. that anyone could simply install and enable. Pre-processing images to dynamically poison them on each request should be fun, and also protect against a deduplication defense. I'd certainly install that.
Re: AI scrapers request commented scripts
#187>These were scrapers, and they were most likely trying to non-consensually collect content for training LLMs. "Non-consensually", as if you had to ask for permission to perform a GET request to an open HTTP server. Yes, I know about weev. That was a travesty.
If you're lying in the requests you send, to trick my server into returning the content you want, instead of what I would want to return to webscrapers, that's non-consensual. You don't need my permission to send a GET request, I completely agree. In fact, by having a publicly accessible webserver, there's implied consent that I'm willing to accept reasonable, and valid GET requests. But I have configured my server t…
Somebody concealing or obfuscating various information a browser would send by standard for privacy or other reasons is also “lying” by that standard? Or someone using a VPN?
Re: AI scrapers request commented scripts
#188Earlier quoted context omitted.
> Looks like it's time for in-browser scrappers. If scrapers were as well-behaved as humans, website operators wouldn't bother to block them[1]. It's the abuse that motivates the animus and action. As the fine articles spelt out, scrapers are greedy in many ways, one of which is trying to slurp down as many URLs as possible without wasting bytes. Not enough people know about common crawl, or know how to write multith…
The question is who runs them? There are only a few big companies like MS, Google, OpenAI, Anthropic. But from the posts here it looks like hordes of buggy scrapers run by enthusiasts.
Ad companies, even the small ones, "Brand Protection" companies, IP lawyers looking for images that were used without license, Brand Marketing companies, where it matters also your competitors etc etc
Re: AI scrapers request commented scripts
#189[flagged]
Yep. Robots.txt is a framework intended for performance, not a legal or ethical imperative. If you want to control how someone accesses something, the onus is on you to put access controls in place. The people who put things on a public, un-restricted server and then complain that the public accessed it in an un-restricted way might be excusable if it's some geocities-esque Mom and Pop site that has no reason to know…
What controls do you suggest?
Saying that a handful of mass copyright infringers with billion dollar investors are simply part of the "public" like every regular visitor is seriously distorting the issue here.
Sites with a robots.txt banning bots are only "unrestricted" in a strictly technical sense. They are clearly setting terms of use that these rogue bots are violating. Besides, robots.txt is legally binding in certain jurisdictions, it's not just a polite plea. And if we decide that anything not technically prevented is legal, then we're also legitimising botnets, DDoS attacks, and a lot more. Hacking into a corporate system through a malconfiguration or vulnerability is also illegal, despite the fact that the defenses failed.
Finally, we all know that the only purpose these bots are scraping for is mass copyright infringement. That's another layer where the "if it's accessible, it's fair game" logic falls apart. I can download a lot of publicly accessible art, music, or software, but that doesn't mean I can do with those files whatever I want. The only reason these AI companies haven't been sued out of existence yet, like they should've been, is that it's trickier to prove provenance than if they straight up served the unmodified files.
Re: AI scrapers request commented scripts
#190Earlier quoted context omitted.
When I open an HTTP server to the public web, I expect and welcome GET requests in general. However, (1) there's a difference between (a) a regular user browsing my websites and (b) robots DDoSing them. It was never okay to hammer a webserver. This is not new, and it's for this reason that curl has had options to throttle repeated requests to servers forever. In real life, there are many instances of things being off…
> robots.txt. This is not the law In Germany, it is the law. § 44b UrhG says (translated): (1) Text and data mining is the automated analysis of one or more digital or digitized works to obtain information, in particular about patterns, trends, and correlations. (2) Reproductions of lawfully accessible works for text and data mining are permitted. These reproductions must be deleted when they are no longer needed for…
I doubt robots.txt would fit. robots.txt allows or disallows access, but it does not state any claim. You can license content you don't own, put it on your website, and then exclude it in robots.txt without that implying any claims of rights to that content.