How about making cgit more efficient at serving these pages. There's no excuse for burning a ton of CPU power on purely static pages when you have generous resources available to you.
Creepy Crawlies
491–500 of 684 posts
Re: Creepy Crawlies
#492Re: Creepy Crawlies
#493Earlier quoted context omitted.
Exactly my observation as well. They devour absolutely everything, no exceptions. No matter how stupid it might be to digest a source code repository via HTTP. They probably don't even recognize what's inside those pages and that there's an easier way to obtain the same result.
The crawlers are not AI. The crawlers are deterministic. They are collecting data to train AIs.
But gitweb is probably the second most used method of hosting a git repo and easily recognizable through heuristics. If it’s gitweb, fallback to git access and save everyone, including the crawler, time and resources.
Re: Creepy Crawlies
#494These are most likely not training scrapers, but people looking for concrete pieces of information (i.e. commit, comment, etc).
Re: Creepy Crawlies
#495Earlier quoted context omitted.
I think you underestimate the difficulty in effectively gating user registration. If the anti-bot efforts in general don't work for other site pages, they won't work for signup functionality either.
I've been on the other side of this kind of thing (hero rather than villain, though I'm sure someone out there disagrees). You can fairly arbitrarily increase the difficulty of user registration, far beyond what users will tolerate for viewing an individual page. You can exploit this; a bot needs to make many accounts for the activity desired, and it's not hard to make "generate account" more expensive than it's wort…
What if you make MFA mandatory. Normal users should have little issues with that despite the increased friction but bots would struggle with this.
Re: Creepy Crawlies
#496Earlier quoted context omitted.
> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won. You yourself describe the solution. Extensions or browser integrations of the Anubis task.
He didn’t describe a solution. He described a (crappy) workaround for humans. But the fact is that this cannot and will not stop bots. The people running bots can do the same, even faster.
He did in fact describe it.
Re: Creepy Crawlies
#497Earlier quoted context omitted.
I had a fairly long conversation with Xe privately about this and didn't feel the need to repeat it here, especially since (as the search box will tell you) I've already talked about this before. But, do go on.
I looked at your comments and there's a bunch saying that Anubis is bad CS but none I could find saying what, exactly, you want Anubis to do instead. You sure are posting a lot about how it's bad, though.
It is interesting, this particular user seems extremely active in this thread. I'm not sure if they have a bone to pick with Anubis in particular or some other reason for disliking the software so much, but to me it does seem overzealous. Regardless of if it's "bad computer science" or what, Anubis works quite well and is easy to customise to target it to specific botnets, as I've done. And this is coming from someone trying to develop her own non-PoW anti-scraper system.
Re: Creepy Crawlies
#498Earlier quoted context omitted.
1. Preventing access to open-source code by scrapers is not the goal behind this idea. The goal is to reduce the server load taken by the scrapers. 2. Why would you need a full clone to access /blob/cee9395acd8043be0644b25c34bfa86623f2b935/block/badblocks.c?
I don't think the show-commit or file-at-revision routes are what's causing the bot load. Rather, the article describes the routes that compute views (e.g. show the change history across several commits) into history as being the issue. If we take that example, a JS client would have to fetch N individual commits/routes/objects and compute the requested view, but first it'd have to fetch the indexes/logs to determine…
This is a much more substantial critique :) I definitely don't think this is a trivial idea. I did a quick analysis and the naivest possible solution isn't very good: only looking at uncompressed objects requires scanning ~340 MB to deliver the But this begs the question, why is cgit scanning 340 MB to answer this question? Is cgit not using good indexes? (Making a space-time tradeoff that doesn't make sense on a public-access website in a post-stupid-AI-scraper world?)
Re: Creepy Crawlies
#499I made it to a third round interview at anthropic in 2024 and they had me build a web crawler as their programming test. Part way through I started on making it respect robots.txt and I could immediately tell they were no longer interested in me.
That's really funny. It's always been kind of amusing to me that Anthropic has this air about them of trying to be the most ethical AI company, but really exhibits the same behaviour as all the others.
Re: Creepy Crawlies
#500I maintain a formerly popular gaming website, and it used to have hundreds of legitimate requests per second. The load would be especially high during popular event times. So, it’s always been running on a dedicated server. It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and d…
> Overall, it seems like the LLM are really straining the internet economy, the openness of it I feel like it's the same issue that public resources are facing. Step 1: public resource is freely available Step 2: asshole shows up, drains the resource completely just because they can Step 3: either resource is taken away, or it has massive security behind it