Amazon's AI crawler is making my Git server unstable

How many TB is your repo?

Do they keep retrieving the same data from the same links over and over and over again, like stuck in a forever loop, that runs week after week?

Or are they crawling your site at a hype aggressive way but getting more and more data? So it may tea them last say 2 days to crawl over it and then they go away?

Depends on what you are hosting. I found that source code repository viewers in particular (OP mentions Gitea, but I have seen it with others as well) are really troublesome: Each and every commit that exists in your repository can potentially cause dozens if not hundres of new unique pages to exist (diff against previous version, diff against current version, show file history, show file blame, etc...). Plus many repo viewers of them take this information directly from the source repository without much caching involved, as it seems. This is different from typical blogging or forum software, which is often designed to be able to handle really huge websites and thus have strong caching support. So far, nobody expected source code viewers to be so popular that performance could be an issue, but with AI scrapers this is quickly changing.

> What if instead it was possible to feed the bots clearly damaging and harmfull content?

With all respect, you're completely misunderstanding the scope of AI companies' misbehaviour.

These scrapers already gleefully chow down on CSAM and all other likewise horrible things. OpenAI had some of their Kenyan data-tagging subcontractors quit on them over this. (2023, Time)

The current crop of AI firms do not care about data quality. Only quantity. The only thing you can do to harm them is to hand them 0 bytes.

You would go directly to jail for things even a tenth as bad as Sam Altman has authorized.

Not a crawler writer but have FAFOd with data structures in the past to large career success.

...

The closest you could possibly do with any meaningful influence, is option C, with the general observations of:

1. You'd need to 'randomize' the generated output link

2. You'd also want to maximize cachability of the replayed content to minimize work.

3. Add layers of obfuscation on the frontend side, for instance a 'hidden link (maybe with some prompt fuckery if you are brave) inside the HTML with a random bad link on your normal pages.

4. Randomize parts of the honeypot link pattern. At some point someone monitoring logs/etc will see that it's a loop and blacklist the path.

5. Keep up at 4 and eventually they'll hopefully stop crawling.

---

On the lighter side...

1. do some combination of above but have all honeypot links contain the right words that an LLM will just nope out of for regulatory reasons.

That said, all above will do is minimize pain (except, perhaps ironically, the joke response which will more likely blacklist you but potentially get you on a list or a TLA visit)...

... Most pragmatically, I'd start by suggesting the best option is a combination of nonlinear rate limiting, both on the ramp-up and the ramp-down. That is, the faster requests come in, the more you increment their 'valueToCheckAgainstLimit`. The longer it's been since last request, the more you decrement.

Also pragmatically, if you can extend that to put together even semi-sloppy code to then scan when a request to a junk link that results in a ban immediately results in another IP trying to hit the same request... well ban that IP as soon as you see it, at least for a while.

With the right sort of lookup table, IP Bans can be fairly simple to handle on a software level, although the 'first-time' elbow grease can be a challenge.

Why work hard… Train a model to recognize the AI bots!