551–560 of 694 posts
Side note: why are shallow clones evil? I always thought they were cheaper, but I guess that’s really just for my disk space. (since the server has to compute what blobs to give you instead of just “everything”?)
It's the point that surprised me the most! We always used shallow clones, to speed the CI, I didn't knew that it got that much impact server side!
Earlier quoted context omitted.
I looked at your comments and there's a bunch saying that Anubis is bad CS but none I could find saying what, exactly, you want Anubis to do instead. You sure are posting a lot about how it's bad, though.
It is interesting, this particular user seems extremely active in this thread. I'm not sure if they have a bone to pick with Anubis in particular or some other reason for disliking the software so much, but to me it does seem overzealous. Regardless of if it's "bad computer science" or what, Anubis works quite well and is easy to customise to target it to specific botnets, as I've done. And this is coming from someon…
(Nothing against it, I'm always happy to hear what Thomas has to say)
Earlier quoted context omitted.
Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them Fo…
> Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them Depending on the configuration, Anubis will supply a token after the challenge that bypasses the challenge for a time. So any scraper that retains basic cookies will be able to bypass the challenge for a number of page views. A user who needs to load a single page and…
A legitimate user solves it once, shows legitimate-use behavior, and retains their access for thousands of requests. A crawler solves a challenge, is determined to use bot-like behavior after a dozen requests, gets an IP ban, and needs to recycle their IP and start from scratch.
It is indeed worse for fly-by users, but with a website like Linux cgit those basically don't exist. It would be a completely different story for, say, someone's personal blog.
If so, you could serve iocaine-style bogus pages 1% (say) of the time that:
1. "Look like" real pages to an LLM-less computer (if you get to the point where you have pushed crawlers to use LLMs to detect nonsense, that already increases the cost a lot)
2. Look "obviously wrong" to a human (E.g., you could take some regular text and swap the order of each adjacent pair of words)
3. Are cheap to generate
4. Important: Contain more links than regular pages, on average, and each to an always-bogus page
The idea is that, due to the large number of pages fetched by crawlers, even with a very low "random bogus page rate", like 1%, they will soon unwittingly hit a bogus page, from which point the fraction of their time spent accessing expensive genuine pages will fall exponentially due to the compounding effect of the higher outbound link count on bogus pages. Humans seeing a bogus page will be confused and annoyed, but simply refreshing the page in the browser will solve the problem 99% of the time (and of course the possibility of this happening can be documented, even on the page itself).
The main advantage is that this does not require any IP-based tracking. You could of course decide to apply this only to pages that are already slightly suspicious (e.g., very old commits).
Earlier quoted context omitted.
If you want to argue that there's excessive enthusiasm about anubis, which is neither a permanent solution nor a solution against all bots run for all purposes (some bot authors are cleverer than others, and you don't have to block everything, just enough to reduce resource use to tolerable levels)... fine. However, the refrain from naysayers isn't that there's too much enthusiasm. It's that anubis doesn't make sense…
I'm not moralizing. The sites running Anubis are owned by people who like Anubis and are fully within their rights to run Anubis. I don't care how enthusiastic they are about it. It's their call. I'm just saying, it's bad computer science.
If you were dealing with a DoS like this and had an option that cut the annoying traffic in half in a fairly deterministic way, that sounds like a decent thing to do operationally in the moment right?
> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…
Do you have an underpowered TV box that would need hours to solve difficulty 6? Do you also have a Windows box where somebody installed gta6_leak_full.iso? Great, have the TV send their challenges to the Windows box for solving.
If you're operating at scale, you probably have hyper-optimized Anubis kernels, E.G. compiled for the particular set of SIMD extensions that a given machine has. At that scale, getting 1% more juice out of your machine pool becomes worth the squeeze. In the worst case scenario, you can probably get some cloud boxes on AWS to do the challenge solving for you.
Earlier quoted context omitted.
One possible way to do this: rather than simply burning hashes for nothing, Anubis can burn hashes to mine crypto. This way, the bots pay for the infrastructure they consume, at least in part. Combine with heuristics like in the article, so you can crank up the difficulty only for likely bot traffic, and you've got a bot supported site.
Disappointed at the lack of replies here, I've had a similar thought before and I'd be curious to hear about the technical viability of this
[1] https://krebsonsecurity.com/2018/03/who-and-what-is-coinhive...
I have the feeling that the hate might be misplaced. For a shopping website or user generated content website, I might understand the terrible load of crawlers that are trying to "steal" the data. But for the kernel, what's the purpose? Are you that "no human" are seeing your page or its content? Maybe we should investigate more the usage being this "bots". I don't buy the explanation that there are millions LLM that…
> the C10k challenge is solved since a long time This has nothing to do with that. Any Node.JS application will happily accept 100K connections. They'll all wait for the under-resourced database behind it. That application "solved" the C10K challenge, but it's still overwhelmed. > it might be possible that the "LLM agent" is requesting the commit contents to "understand" or refer or explain them. Is it a bad thing if…
> But most of the routes scrapers are hitting are e.g. computing diffs between arbitrary pairs of commits, or other computed-on-the-fly views into history.
Looks like to be wrong based on the article: Today, git.kernel.org receives about 6M daily requests demanding to see random commits.
Also, in "how bad is it", there is a bar graph of the tasks requested. And only diff can't be cached, and it is clearly not the most important one.
> batched waves from laundered residentail proxy IP addresses
Again, I don't see anywhere anything stating that there was "grouped" requests that are directly related, except in something not related to AI: The thing that usually takes us down are not scraper bots, but poorly designed CI systems that try to do something stupid like shallow-clone stable.git from 20 different nodes, all at the same time. (Shallow clones are awful. Run your own damn mirror if you're going to do something nasty like that.)
Just "waves" of requests coming from the same "residential proxy IP". But that makes sense because they might come from a same service without being a single "source" trying to scrap for training.
Let's say that I'm chatgpt, or Claude, and I have a lot of user related requests to do things based on the commits there, each one will be a different task, but they will all go through the same pipe, that would be a proxy with the residential IP if the provider notice major websites like this one blocking requests for bots.
Earlier quoted context omitted.
Monero is fine with making people wait, that's literally the whole basis of mining blocks in "crypto currency". It's tuned to the time required. Anubis wants "slow for bots but fast for humans" which is not possible if somebody has a farm of devices that costs them basically zero.
One possible way to do this: rather than simply burning hashes for nothing, Anubis can burn hashes to mine crypto. This way, the bots pay for the infrastructure they consume, at least in part. Combine with heuristics like in the article, so you can crank up the difficulty only for likely bot traffic, and you've got a bot supported site.