> Training an LLM on content produced by the LLM gives it the equivalent of a digital prion disease
Is it foolish of me to have expected more from a blog post on kernel.org?
Creepy Crawlies
31–40 of 687 posts
Re: Creepy Crawlies
#32I maintain a formerly popular gaming website, and it used to have hundreds of legitimate requests per second. The load would be especially high during popular event times. So, it’s always been running on a dedicated server. It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and d…
What is "formally popular"?
Re: Creepy Crawlies
#33High Anubis difficulty is annoying the hell out of me for several sites. And it's starting to not block LLM bots anymore? > 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
In a few years the VC money will dry up and this gross overspend on slurping data will end.
Re: Creepy Crawlies
#34How expensive would AI access be if every user paid their fair share instead of shoving it on the people doing the actual work?
See also: the price for uber rides.
Re: Creepy Crawlies
#35I maintain a formerly popular gaming website, and it used to have hundreds of legitimate requests per second. The load would be especially high during popular event times. So, it’s always been running on a dedicated server. It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and d…
What is "formally popular"?
From context I'd say they meant "formerly" and that's just a typo. I didn't even notice the error before you pointed it out.
Re: Creepy Crawlies
#36couldn't you have anubis on a dynamic difficulty? ie if a ip requests more than 1k pages per day +1 the difficulty ?
Maybe anubis difficulty should depend on the age of commit. This year - 4, everything older 8
Re: Creepy Crawlies
#37Side note: why are shallow clones evil? I always thought they were cheaper, but I guess that’s really just for my disk space. (since the server has to compute what blobs to give you instead of just “everything”?)
Re: Creepy Crawlies
#38Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.
The issue is there's far more pages than could reasonably be stored: cgit is an old-school server-side renderer and there's a combinatorial explosion of possible pages it could render. The article calculates there's about a quadrillion different pages that could be 'crawled'
Re: Creepy Crawlies
#39> Training an LLM on content produced by the LLM gives it the equivalent of a digital prion disease Is it foolish of me to have expected more from a blog post on kernel.org?
Are you trying to say that's bad writing? I think it's a good metaphor for a documented phenomenon: https://en.wikipedia.org/wiki/Model_collapse
Re: Creepy Crawlies
#40But why don't they just git clone?
These crawlers (in contrast to e.g. googlebot and similar better behaved crawlers) are not very smart: they seem to make very little effort to avoiding crawling useless deep trees of generated pages. About the only thing they seem to put a lot of effort into is avoiding blocking.
(I'll note that while these are generally attributed to AI data gathering because of the timing of when they took off, it's not actually obvious who's running these bots. The big players all have crawlers that identify themselves and are reasonably well behaved, but I don't know if anyone has managed to positively attribute these other ones to any particular group)