If you cite an article, you should link to it.
FOSS infrastructure is under attack by AI companies
191–200 of 631 posts
Re: FOSS infrastructure is under attack by AI companies
#192The days of an open web are long gone. Every server will eventually have to require authentication for access, and to get an account you will have to provide some form of payment or social proof. Honestly, I don't see it necessarily as a bad thing.
If that was piggy-backed on ActivityPub, Matrix, Solid, or something else decentralised, and if I could say "this bot is acting as my agent, if it misbehaves then I personally get blocked" then there could be something in this. I don't see how to get around artificial identity farms though. That's also not something that payment or social proof fixes. If payment isn't trivial then you exclude genuine people; if it's the act of interacting with a payment processor that's being taken as proof-of-existence then it's outsourcing the ability to interact with anything in the modern world to Visa and Mastercard. That's bad. Social proof is also problematic because if your business is to run an identity farm, then having all your identities interact in legible ways isn't hard, so the social proof needs to be grounded in something global, and there are approximately no good choices.
Re: FOSS infrastructure is under attack by AI companies
#193To be clear, this is not an attack in the deliberate sense, and has nothing to do with AI except in that AI companies want to crawl the internet. This is more "FOSS sites damaged by extreme incompetence and unaccountability." The crawlers could just as well be search engine startups.
Yes, any crawler or some traffic peak can do this if your infrastructure is not well engineered
You're getting it the wrong way. It's: Any crawler that's not well engineered, that doesn't follow robots.txt, that fakes its User Agent, that doesn't allow you to contact hem, that fetches content an indiscriminate number of times, repeatedly, all day long... can do this to your infrastructure unless you're a giant.
What these crawlers are doing is akin to DDoS attacks.
Re: FOSS infrastructure is under attack by AI companies
#194I see this going the way of email, with larger, well-known, and more well-behaved crawlers being allowed to index websites for free, and smaller, unknown crawlers suffering brownouts, getting banned, or having to pay for access. It will be harder to self-host a website or run your own crawler.
Re: FOSS infrastructure is under attack by AI companies
#195there is simple solution for gentlemens: robots.txt should allow to exclude all AI crawlers and AI crawlers should be forced to add "AI" to their crawl user agent headers and also respect robots.txt saying they can't crawl this website right now we need to do this: User-agent: * Disallow: /
Nice. How do we force them to respect robots.txt?
Re: FOSS infrastructure is under attack by AI companies
#196> According to Drew, LLM crawlers don't respect robots.txt requirements and include expensive endpoints like git blame, every page of every git log, and every commit in your repository. They do so using random User-Agents from tens of thousands of IP addresses, each one making no more than one HTTP request, trying to blend in with user traffic. How do they know that these are LLM crawlers and not anything else?
As someone that is also affected by this: We see a manifold increase in requests since this LLM crap is going on. Many of these IPs come from companies that obviously work with LLM technology, but the problem is that it's 100s of IPs doing 1 request, not 1 IP doing 100s of requests. It's just extremely unlikely that anyone else is responsible for this.
Re: FOSS infrastructure is under attack by AI companies
#197Earlier quoted context omitted.
I don’t think you’re missing anything, I think the plan really is to burn it all down and rule over the ashes. The old saw “if you’re so smart, why aren’t you rich?” works in reverse too. This is a foolish, shortsighted thing to do, and they’re doing it anyway. Not really thinking about where value actually comes from or what their grandchildren’s lives would be like in such a world.
Capitalism is an unthinking, unfeeling force. The writing is on the wall that AI is coming, and being altruistic about it doesn’t do jack to keep others from the land grab. Their thinking is, might as well join the rush and hope they’re one of the winners. Every one of us sitting on the sidelines will be impacted in some way or the other. So who’re the smart ones, the ones who grab shovels and start digging, or the o…
I already started incorporating AI into my workflow. It's definitely helped with productivity.
At some point in the future, if you aren't using AI, you won't be able to compete in the job market.
Re: FOSS infrastructure is under attack by AI companies
#198Earlier quoted context omitted.
Hi! I do this! See https://github.com/TecharoHQ/anubis for more info!
I hope lots of websites adopt this, mainly because I want to see more happy jackal girls while browsing.
My monetization strategy is unironically to offer a de-anime'd version under the name Techaro BotStopper or something.
Re: FOSS infrastructure is under attack by AI companies
#199Earlier quoted context omitted.
Yep. And it is much more far reaching than that. Look at the primary economic claim offered by AI companies: to end the need for a substantial portion of all jobs on the planet. The entire vision is to remake the entire world into one where the owners of these companies own everything and are completely unconstrained. All intellectual property belongs to them. All labor belongs to them. Why would they need good will…
I have this line of thought as well but then I wonder, if we are all out of jobs and out of substantial capital to spend, how do these owners make money ultimately? It's a genuine question and I'm probably missing something obvious. I can see a benevolant/post-scarcity spin to this but the non-benevolant one seems self defeating.
Unfortunately I don’t think the logic extends beyond “if we don’t do it, someone else will”. Anything after that is secondary.
Re: FOSS infrastructure is under attack by AI companies
#200Earlier quoted context omitted.
I have this line of thought as well but then I wonder, if we are all out of jobs and out of substantial capital to spend, how do these owners make money ultimately? It's a genuine question and I'm probably missing something obvious. I can see a benevolant/post-scarcity spin to this but the non-benevolant one seems self defeating.
Money is only a bookkeeping tool for complex societies. The aim of the owner class in a worker-less world would be accumulation of important resources to improve their lives and to trade with other owners (money would likely still be used for bookkeeping here). A wealthy resource-owner might strive to maintain a large zone of land, defended by AI weaponry, that contains various industrial/agricultural facilities prod…
Better for them to give us jobs so we owe them and are less likely to revolt!