There are options beyond auth walls for detecting/enforcing behavior as well since these scrapers have very recognizable device signatures: https://stytch.com/blog/detecting-ai-agent-use-abuse/
FOSS infrastructure is under attack by AI companies
431–440 of 631 posts
Re: FOSS infrastructure is under attack by AI companies
#432Earlier quoted context omitted.
I already started incorporating AI into my workflow. It's definitely helped with productivity. At some point in the future, if you aren't using AI, you won't be able to compete in the job market.
At some point in the future, if you aren't AI, you won't be able to compete in the job market.
That's a laughable idea.
Job market is formed by the presence of needs and the presence of the ability to satisfy them. AI - does not reduce the ability to satisfy needs, so only possible situation where you won't be able to compete - is either the socialists will seize power and ban competition, or all the needs will be met in some other ways. In any other situation - there will be job market and the people will compete in it
Re: FOSS infrastructure is under attack by AI companies
#433Earlier quoted context omitted.
Traditionally, holders of IP ranges that attack the internet at large get kicked off the internet by having those ranges blacklisted everywhere. This can also get them in serious trouble with the places they got their IP ranges (I assume AWS has them directly from ARIN, so maybe not) and their upstream bandwidth providers and so on, as well as making them less attractive hosts because they are blocked everywhere.
Kicking AWS of the internet would effectively break half of everything. So much stuff is relying on services running on AWS that it's not even funny. We've seen random stuff break when AWS has had outages, not because we used AWS ourselves, but because suppliers do.
That's actually an argument in favour of kicking AWS off the Internet. We rely too much on their services, to the point we're afraid of banning their IPs if they do something bad. Better stop this now than being worse off later. The best moment would have been ten years ago, the second best moment is today.
Re: FOSS infrastructure is under attack by AI companies
#434We need a project in the spirit of Spamhaus to actively maintain a list of perpetrating IPs. If they're cycling through IPs and IP blocks I don't know how sustainable a CAPTCHA-like solution is.
This sounds backwards to me, if you maintain a list of IPs but they are constantly cycling them, it'll get out of date quickly, but a captcha-like system will (hopefully) always stop bot traffic
While some of the residential IPs are from malware, a lot of it is from residential IP proxies, where people are paid to run proxy software from their home. If it starts getting around that people who run this software quickly become blocked by the majority of the internet that will lessen that part of the problem.
Re: FOSS infrastructure is under attack by AI companies
#435Earlier quoted context omitted.
If that was piggy-backed on ActivityPub, Matrix, Solid, or something else decentralised, and if I could say "this bot is acting as my agent, if it misbehaves then I personally get blocked" then there could be something in this. I don't see how to get around artificial identity farms though. That's also not something that payment or social proof fixes. If payment isn't trivial then you exclude genuine people; if it's…
It doesn't have a global solution and it doesn't need to be implemented only on a specific technology-based system. I mean, at Communick I offer Matrix, Mastodon, Funkwhale and Lemmy accounts only for paying customers. As such, I have implemented payments via Stripe for convenience, but that didn't stop from getting customers who wanted to pay directly via crypto, SEPA and even cash. It also didn't stop me from bypas…
Right, so the gap I'm seeing is that anyone who wants to identity farm just does what you've done. The problem isn't the assurance that you receive, it's the assurance you give to anyone else that any of your customers are flesh-and-blood.
Re: FOSS infrastructure is under attack by AI companies
#436Earlier quoted context omitted.
Author of Anubis here. If that happens, I win.
If that happens, count with me to use Anubis to factor large primes or whatever science needs as a background task.
Actually, that is not a bad idea. @xena maybe Anubis v2 could make the client participate in some sort of SETI@HOME project, creating the biggest distributed cluster ever created :-D
Re: FOSS infrastructure is under attack by AI companies
#437Isn't this just poor, sloppy crawler implementation? You shouldn't need to fetch a repo more than once to add it to a training set.
From reading Drew Devaults angry post from earlier this week, my take is that not only is it poorly implemented crawlers, it's also that it's cheaper to scrape, rather than keep copies on hand. Effectively these companies are outsourcing the storage of "their" training data to everyone on the internet. Ideally a site would get scraped once, and then the scraper would check if content has changed, e.g. etag, while als…
That seems crazy - millions of $ on GPUs but they can't afford some cheap storage? And direct network scraping seems super high latency. Although I guess a massive pretaining run might cycle through the corpus very slowly. Dunno, sounds fishy.
Re: FOSS infrastructure is under attack by AI companies
#438Would it be possible to use security flaws in their crawler to make them do very expensive things? In the case of China, illegal (anti CCP) things? I don't think many of those companies are too rigorous about their cybersecurity, and cybersecurity is expensive per se anyway.
Re: FOSS infrastructure is under attack by AI companies
#439Earlier quoted context omitted.
I already started incorporating AI into my workflow. It's definitely helped with productivity. At some point in the future, if you aren't using AI, you won't be able to compete in the job market.
I started a borderline smug response here pointing out how bullshit white collar and service jobs* where in deep shit but folks who actually work for a living would be fine. I scrapped it halfway through when it occurred to me that if everyone's broke then by definition nobody's spending money on stuff like contractors, mechanics, and other hardcore blue collar trades. Toss in AI's force multiplication of power deman…
>if everyone's broke
>nobody's spending money on stuff like contractors, mechanics, and other hardcore blue collar trades.
Why would this happen? Money is simply a medium of exchange of values that this contractors, mechanics and other hardcore blue collar trades are creating. How can they be broke, if Ai doesn't disturb their ability to create values and exchange it?
Re: FOSS infrastructure is under attack by AI companies
#440Earlier quoted context omitted.
Yeah, I definitely don't want to pivot this thread into a product pitch, as the important thing is helping the open-source projects, but we can work with the maintainers to tune the systems to be as strict/lax as preferred. I'm sure the other services can too, to be fair.
The underlying issue is that many sites aren't going to get feedback from the real people they've blocked, so their operators won't actually know that tuning is required (also, the more strict the system, the higher percentage of requests will be marked as bots, which might lead an operator to want things to be even more strict...)
I will say -- a higher-end bot detection service should provide paper trails on the block actions they take (this may not be available for freemium tiers, depending on the vendor).
But to your point, the real kicker is the "many sites aren't going to get feedback from the real people they've blocked" since those tools inherently decided that the traffic was not human. You start getting into Westworld "doesn't look like anything to me" territory.