To be clear, this is not an attack in the deliberate sense, and has nothing to do with AI except in that AI companies want to crawl the internet. This is more "FOSS sites damaged by extreme incompetence and unaccountability." The crawlers could just as well be search engine startups.
> "FOSS sites damaged by extreme incompetence and unaccountability."
Given this info, the natural next question is “who is doing the harm?”
The answer is “AI companies”. Most people would now view the situation as having a lot to do with AI companies.
Might be time for a class action lawsuit. Something like that could work out really well for the little guy as it would probably make a dent in the LLM companies pockets and access to data.
I've pointed this out to a few people in this space. They tend to suggest that the value in AI is so great this means we should get rid of copyright law entirely.
That value is only great if it's shared equitably with the rest of the planet. If it's owned by a few, as it is right now, it's an existential threat to the life, liberty, and pursuit of a happiness of everyone else on the planet. We should be seriously considering what we're going to do in response to that threat if something doesn't change soon.
> That value is only great if it's shared equitably with the rest of the planet.
I think this should be an axiom which should be respected by any copyright rule.
How would those 5 lines of code look like? The base of this solution is that it asks to solve a computationally-intensive problem whose solution, once provided, isn't computationally-intensive to check. How would those 5 lines of code change this?
> Look at the primary economic claim offered by AI companies: to end the need for a substantial portion of all jobs on the planet. And this is why AI training is not "fair use". The AI companies seek to train models in order to compete with the authors of the content used to train the models. A possible eventual downfall of AI is that the risk of losing a copyright infringement lawsuit is not going away. If a court d…
> The AI companies seek to train models in order to compete with the authors of the content used to train the models. When I read someone else’s essay I may intend to write essays like that author. When I read someone else’s code I may intend to write code like that author. AI training is no different from any other training. > If a court determines that the AI output you've used is close enough to be considered a de…
If you as an individual recognizably regurgitate the essay you read, then you have infringed. If an AI model recongnizably regurgitates the essay it trained on then it has infringed. The AI argument that passing original content through an algorithm insulates the output from claims of infringement because of "fair use" is pigwash.
Yep -- our story here: https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse... (quoted in the OP) -- everyone I know has a similar story who is running large internet infrastructure -- this post does a great job of rounding a bunch of them up in 1 place. I called it when I wrote it, they are just burning their goodwill to the ground. I will note that one of the main startups in the space worked with us direct…
>which I then emailed 3x and never got a reply. Send a bill to their accounts payable team instead.
Detect AI scraper and inject an in-page notice that by continuing they accept your terms of use.
Terms of use charges them per page load in some terminology of abuse.
Insane, I wonder if we eventually end up with a non-search-engine indexed version of the web that's more like browsing in the 90s where websites just had to link to oneanother to get noticed . . . . I love that the solution to LLM scraping is to serve the browser a proof of work, before they allow access - I wonder if things like new sites start to do this . . . It would mean they won't be indexed by search engines,…
Hi! I do this! See https://github.com/TecharoHQ/anubis for more info!
Does the PoW make money via crypto mining? Or is it just to waste the caller's CPU cycles? If you could monetize the PoW then you could re-challenge at an interval tuned so that the caller pays for their usage.
How are the crawlers identifying themselves? If it's user agent strings then they can be faked. If it's cryptographically secured then you create a situation where newcomers can't get into the market.
Google publishes the ip addresses that google bot uses. If someone claims to be google bot but is not from one of those addresses, it’s a fake.
I don't see how that helps the case where the UA looks like a normal browser and the source IP looks residential.
This article starts by citing a blog article - displays a screenshot of the article - but doesn't link to it.
Niccolò here, I'm really sorry about that -- I'm using a weird tooling system to handle articles, which currently has issues with links. I'm working to fix that asap.
screenshots in place of links, cited articles likely shadow-banned on HN, FOSS is under attack 2.0.