My question is, can we serve PoW challenges to these AI LLM scrapers that can be profitable?
I know that mCaptcha is based on PoW. It may be usable here.
FOSS infrastructure is under attack by AI companies
141–150 of 631 posts
Re: FOSS infrastructure is under attack by AI companies
#142Insane, I wonder if we eventually end up with a non-search-engine indexed version of the web that's more like browsing in the 90s where websites just had to link to oneanother to get noticed . . . . I love that the solution to LLM scraping is to serve the browser a proof of work, before they allow access - I wonder if things like new sites start to do this . . . It would mean they won't be indexed by search engines,…
Hi! I do this! See https://github.com/TecharoHQ/anubis for more info!
Re: FOSS infrastructure is under attack by AI companies
#143Earlier quoted context omitted.
> Copyright the content and sue those who use it for AI training. This costs money, time, and ongoing commitment. FOSS isn't typically known for being overflowing with cash.
The FOSS community can transfer their copyrights to an organisation in order to take legal action against AI operators.
Which organizations are willing to take on that burden (time, expenses, stress, etc.) for free, with no guarantee that they will win?
There is not really any incentive or reward for 3rd-party organizations to step in and do this.
Re: FOSS infrastructure is under attack by AI companies
#144So I'll just float an idea again that always gets rejected here. This is yet another problem that could be solved completely by... Eliminating anonymity by default on the internet. To be clear, you could still have anonymous spaces like Reddit where arbitrary user IDs are used and real identities are discarded. People could opt-in to those spaces. But for most people most of the time, things get better when you can v…
So in this scenario, whose real user ID would be used for the scrapers?
Doesn't this just create an even worse market for identity theft and botnets?
How does this apply to countries without a national ID system like the United States?
What do you do with an ID traced to a different country, anyway?
> personal attacks and threats will be reduced when anonymity is removed
People are happy to make death threats under their real name, newspaper byline, blue tick, or on the presidential letterhead if they're doing so from a position of power.
Re: FOSS infrastructure is under attack by AI companies
#145Yep -- our story here: https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse... (quoted in the OP) -- everyone I know has a similar story who is running large internet infrastructure -- this post does a great job of rounding a bunch of them up in 1 place. I called it when I wrote it, they are just burning their goodwill to the ground. I will note that one of the main startups in the space worked with us direct…
> just burning their goodwill to the ground AI firms seem to be leading from a position that goodwill is irrelevant: a $100bn pile of capital, like an 800lb gorilla, does what it wants. AI will be incorporated into all products whether you like it or not; it will absorb all data whether you like it or not.
I think the logic is more like “we have to do everything we can to win or we will disappear”. Capitalism is ruthless and the big techs finally have some serious competition, namely: each other as well as new entrants.
Like why else can we just spam these AI endpoints and pay $0.07 at the end of the month? There is some incredible competition going on. And so far everyone except big tech is the winner so that’s nice.
Re: FOSS infrastructure is under attack by AI companies
#146The big takeaway here is that Google's (and advertisement in general) dominance over the web is going away. This is because the only way to stop the bots is with a captcha, and this also stops search indexers from indexing your site. This will result in search engines not indexing sites, and hence providing no value anymore. There's probably going to be a small lag as the current knowledge in the current LLMs dry up…
IMO this was one of the real motives for Web Environment Integrity. Allow Google to index but nobody else. We're kind of stuck between a rock and a hard place here. Which do you prefer, entrenched incumbents or affordable/open hosting?
I’m supremely confident that attestation will arrive in one form or another in the near future.
Anonymous browsing and potentially-malicious bots look identical. This was sort of OK up until now.
Re: FOSS infrastructure is under attack by AI companies
#147Yep -- our story here: https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse... (quoted in the OP) -- everyone I know has a similar story who is running large internet infrastructure -- this post does a great job of rounding a bunch of them up in 1 place. I called it when I wrote it, they are just burning their goodwill to the ground. I will note that one of the main startups in the space worked with us direct…
> just burning their goodwill to the ground AI firms seem to be leading from a position that goodwill is irrelevant: a $100bn pile of capital, like an 800lb gorilla, does what it wants. AI will be incorporated into all products whether you like it or not; it will absorb all data whether you like it or not.
Maxim 1: "Pillage, then burn."
Re: FOSS infrastructure is under attack by AI companies
#148I had a similar issue with a Gitea instance that has some public repos. Gitea doesn't follow best practices for a web application and the action to create an archive of a repo is an HTTP GET request instead of an HTTP POST. AI crawlers were hitting that link over and over again causing the server to repeatedly run itself out of disk space.
I'm pretty sure Forgejo had the same issue and it was fixed on their side.
Re: FOSS infrastructure is under attack by AI companies
#149My question is, can we serve PoW challenges to these AI LLM scrapers that can be profitable?
That's what I've been doing! It works shockingly well. https://github.com/TecharoHQ/anubis
Interesting idea. Seems to me it might be possible to use with a Monero mining challenge instead, for those low real traffic applications where most of the requests are sure to be bots.
Re: FOSS infrastructure is under attack by AI companies
#150Earlier quoted context omitted.
The FOSS community can transfer their copyrights to an organisation in order to take legal action against AI operators.
Which organizations are willing to take on that burden (time, expenses, stress, etc.) for free, with no guarantee that they will win? There is not really any incentive or reward for 3rd-party organizations to step in and do this.
That's what GNU was originally for, and why they requested copyright assignment from contributors to their projects.