[deleted]
Creepy Crawlies
61–70 of 686 posts
Re: Creepy Crawlies
#62High Anubis difficulty is annoying the hell out of me for several sites. And it's starting to not block LLM bots anymore? > 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
In a few years the VC money will dry up and this gross overspend on slurping data will end.
Visions of vast data centers surrounded by fields of browning grass, in which aging, rusting, formerly extremely expensive hardware is spending billions of compute cycles looking at anime catgirls
Re: Creepy Crawlies
#63I wonder if one solution here could be to turn up Anubis difficulty if the first page hit is not one of the obvious entry-points to cgit. It could even have a little hint that says something to the effect of “go visit the home page if this is taking too long”. (Better not to ban them entirely, in case people really did click on some random link e.g. in a news story or mailing list message). Distributed scrapers are g…
Almost all of my visits to cgit instances are through direct deep links. Hard to imagine someone randomly browsing git listings
The kernel folks likely have a good profile on what page people trigger Anubis on (i.e. what page people hit first). From that they could make heuristics about what pages are likely to be useful deep links.
Keep in mind that Anubis will rarely inconvenience an actual user who visits the site often; it’s meant to keep out first-time scrapers trying to grab a few pages from their queue.
Re: Creepy Crawlies
#64Tavis Ormandy called this, about Anubis, almost exactly a year ago: https://news.ycombinator.com/item?id=44962529 It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper…
I disagree. The kernel finds it effective - 66% of scrapers are turned away directly. The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site. If each new site they hit forces…
The implication here is that the proxy fridge forwards the Anubis challenge to a dedicated rig controlled by the scraper who efficiently solves it and returns the answer.
Re: Creepy Crawlies
#65Hm, interesting - how will it look the actual solution for such problems in the future. I suppose the issue will continue to grow. First idea - there should be some cost for sending traffic somewhere. And the server owner also should receive pay - not only the internet provider. So, in with this idea, the server owner can potentially increase the amount of computing power to satisfy all requests.
It's a social problem, technical solutions aren't going to work. To solve social problems you need the law to get involved. It should simply be illegal to install malware on someone's TV and use it to hijack their internet connection, and the punishment should be the same as for any theft of service, including imprisonment for people who do it.
Re: Creepy Crawlies
#66I'm assuming they haven't yet sent responses to the bots? Since AI is dumb, you can send errors that tell the bot to git clone rather than crawl. If it's vulnerable to prompt injection, it might listen and do the clone instead and stop trying to solve challenges.
Barring that, I think the solution is to charge money for access. Require users to sign up to render HTML, and provide a form of payment (any form you want). The cost is, say, $0.1 per GB. Rate limit all requests to reduce CPU. For the average user this will cost a few cents. For the bots you'll cover your costs and have a rate limiter to keep your system from being overwhelmed. Or they can git clone for free with no limit.
Re: Creepy Crawlies
#67Tavis Ormandy called this, about Anubis, almost exactly a year ago: https://news.ycombinator.com/item?id=44962529 It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper…
I disagree. The kernel finds it effective - 66% of scrapers are turned away directly. The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site. If each new site they hit forces…
It's weird to believe data center based, Internet-scale scraping operations will be less able to allocate compute to proof-of-work challenges than individual users. This is design problem with things like Anubis: proof-of-work depends on a cost asymmetry between attacker and defender. But in scraping, both legitimate users and scrapers get the same value out of a transaction.
Re: Creepy Crawlies
#68High Anubis difficulty is annoying the hell out of me for several sites. And it's starting to not block LLM bots anymore? > 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
Has anyone considered having Anubis perform more valuable hashing?
Like, maybe you can't stop the LLM bots, but you can use them as one-off Bitcoin pool mining pool participants. You have to assume that making them find hash values with N leading zeroes has led to finding hash values with more than N leading zeroes. Maybe run a Bitcoin node under there and let each visitor take a couple swings for you with their pickaxes.
Re: Creepy Crawlies
#69This is bad enough that I'm going to stop serving cgit. I've been doing cvsweb, then subversion, then cgit over my home server for many many years and for the first time ever this is annoying my own bw usage. It's ridiculous also how you ban an IP then 1 second later another one picks up from where the first one left on.
I had to take down my cgit repository a few months ago. The load was causing other VMs on the same machine to become unusable.
Re: Creepy Crawlies
#70Running Firefox with Temporary Containers Plus makes challenges 10x more annoying :D
(Each new tab is isolated, unless opening a link in a new tab. Same as Safari in private mode)