Creepy Crawlies

Post by desterothx » Mon, Aug 31, 2026, 11:33 AM UTC

Barbing wrote: post #382

zythyx wrote: post #366

Earlier quoted context omitted.

There are definitely ways of proving you're a human, unfortunately it also means giving up your privacy and anonymity (IRL ID Checks combined with appropriate routing and validation - even going as far as certifying the browser being used) Obviously though none of us want to give that up, so the alternative is that we can almost never 'prove' we are human especially with bots getting as smart or smarter than the aver…

Imagine proving you have no financial incentive to get on the whitelist. Reminded of the mules renting Airbnbs to use as USA-based delivery locations (tricking grandma into FedExing cash for one scam or another) - https://getrichslowly.org/scambaiters (probably Jim Browning + Mark Rober specifically https://youtube.com/watch?v=Xvjjpzyiig4 ) But! Using a network of real ID-checked humans to scrape the web, what would…

Post by semiquaver » Mon, Aug 31, 2026, 11:35 AM UTC

inigyou wrote: post #522

afdbcreid wrote: post #449

Earlier quoted context omitted.

But they operate from neither. You can at most sue the one renting them IP addresses. Which will do basically nothing. (Also, blocking a whole country is likely not what you do, but you probably know that).

Why do you think you can only use the one who's renting them IP addresses?

Post by sigbottle » Mon, Aug 31, 2026, 11:47 AM UTC

vintermann wrote: post #235

mzajc wrote: post #201

> Why is git.kernel.org “interesting” to crawlers I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests. The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this mea…

A lot of work is apparently put into bypassing any kind of anti-scraping, no work is apparently put into figuring out if the site freely gives a way to get all that information in a less wasteful way.

Post by andai » Mon, Aug 31, 2026, 11:53 AM UTC

Demiurge wrote: post #119

andai wrote: post #30

Earlier quoted context omitted.

> And I think there is no reason to act like this race really has to be at such a breakneck speed. I would promote this idea to all of my competitors. Nah mate, you don't have to ask her out right now. You can wait until next week ;) Anyway, silver linings, looks like we're finally going to get widely adopted infra for microtransactions. https://web.archive.org/web/20030202042510/http://www.openp2...

I’m not a participant in this race. Are the AI companies worried about anything but their valuations? I don’t care about their valuations, but I do care about the risks that they are creating for the economy, society, and the technological advancement, at large. Micro transactions [in this case] are a great idea, these crawlers need to be taxed and made to pay for the unaccounted external costs. Furthermore, we need…

Post by xena » Mon, Aug 31, 2026, 11:58 AM UTC

LatticeAnimal wrote: post #152

semiquaver wrote: post #126

Earlier quoted context omitted.

OK, but that doesn't change anything. You have a large pool of IPs, each of which only needs to expend a handful of extra milliseconds of work to get unlimited access to the protected resource. Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a…

Some of the asymmetry might be regained if anubis had thousands of variations of PoW algorithms, each different enough that they must be solved independently. I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...