How expensive would AI access be if every user paid their fair share instead of shoving it on the people doing the actual work?
You mean shoving it onto the investors?
Creepy Crawlies
11–20 of 694 posts
Re: Creepy Crawlies
#12[deleted]
Re: Creepy Crawlies
#13[deleted]
Re: Creepy Crawlies
#14Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.
Or, html generation could be very, very fast in the first place. For example, serving static files is usually just as fast as a reverse-proxy cache.
Interestingly enough, cgit bills itself as "a hyperfast web frontend for git repositories" [0]. Alas, nothing is fast enough for the bots, it would appear.
Re: Creepy Crawlies
#15> So, you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right? Clone the repos, walk every commit. Done.
If repos like this were 10%+ of what they crawled, having a special case for clone-able repos would be smart, but if you're crawling everything, you're not going to do an efficiency tweak for each special case that has a more parse-able option.
Re: Creepy Crawlies
#16couldn't you have anubis on a dynamic difficulty? ie if a ip requests more than 1k pages per day +1 the difficulty ?
Re: Creepy Crawlies
#17Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.
The cache gets evicted between the times the old commits are revisited.
Re: Creepy Crawlies
#18But why don't they just git clone?
Re: Creepy Crawlies
#19Off-topic, but anyone with which he did the plots?
Re: Creepy Crawlies
#20But why don't they just git clone?
Because they're crawling a billion webpages, only a tiny fraction of which can be git cloned, and configuring a special case just for that tiny fraction isn't worth the effort (of the crawlers).