Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

381–390 of 698 posts

Re: Creepy Crawlies

#381

Post by mkl »

ivanjermakov wrote: post #359

There has to be some not-yet-discovered way to have a capcha that is easy for any human but impossible for robot. Too bad capchas hurt user experience no matter how easy they are. Another solution I came up with while reading HN comments: whitelist IPs instead blacklisting. Give access to well-behaving hosts/groups. It can even be shared across different sites. Although this would create a market for selling "good IP…

> There has to be some not-yet-discovered way to have a capcha that is easy for any human but impossible for robot.

Why? We are not so special or magical.

Re: Creepy Crawlies

#382

Post by Barbing »

zythyx wrote: post #366
ivanjermakov wrote: post #359

There has to be some not-yet-discovered way to have a capcha that is easy for any human but impossible for robot. Too bad capchas hurt user experience no matter how easy they are. Another solution I came up with while reading HN comments: whitelist IPs instead blacklisting. Give access to well-behaving hosts/groups. It can even be shared across different sites. Although this would create a market for selling "good IP…

There are definitely ways of proving you're a human, unfortunately it also means giving up your privacy and anonymity (IRL ID Checks combined with appropriate routing and validation - even going as far as certifying the browser being used) Obviously though none of us want to give that up, so the alternative is that we can almost never 'prove' we are human especially with bots getting as smart or smarter than the aver…

Imagine proving you have no financial incentive to get on the whitelist.

Reminded of the mules renting Airbnbs to use as USA-based delivery locations (tricking grandma into FedExing cash for one scam or another) - https://getrichslowly.org/scambaiters (probably Jim Browning + Mark Rober specifically https://youtube.com/watch?v=Xvjjpzyiig4 )

But! Using a network of real ID-checked humans to scrape the web, what would that be--half a billion times harder than Firecrawl or whatever they use today?

Too bad it's dead in the water today because so many (like me) hate the idea so much. Perhaps a biometric dongle (retinal-scanning orb :-/ ) that the staunchest privacy hawks stamp with their seals of approval because it's somehow nearly impossible to go horribly horribly... horribly... wrong... Yeah, anybody who can crack this issue, hope you have the free time or find the funding to try it, we need ya.

Re: Creepy Crawlies

#383

Post by Waterluvian »

colinsane wrote: post #330
Waterluvian wrote: post #75

> It was immediately extremely effective — the bots just gave up. For a few months, it was bliss: bots were blocked at the perimeter and gave up, moving on to easier targets; the users were mildly annoyed but tolerated it, and the Anubis stack was easy enough to deploy everywhere. As a tech person who works with tech people, I have become extremely sensitive to this kind of bias. Is this solution actually better? Is…

> Is the CPU cost actually worse than mildly annoying everyone > What is the cost of 20% and is that cost worth “mildly annoying” everyone? from the articled: > With a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic — everything else are scrapers. this is not some "CPU use is 20% higher than baseline" situation. it seems that people still do not understand the scale of t…

Ah yeah I was quoting 20% of their total capacity.

Nevertheless, without knowing what it actually is in cost, it’s impossible to have an objective sense of the severity of this problem.

Is it, “everyone’s mildly annoyed so we can save $10k per year” or is it “everyone’s mildly annoyed so we can save $100k per year?” And has someone attempted to reason about what the $/year cost is of that mild annoyance?

I’ve seen this a lot in my career. Developers coming to conclusions that are ultimately based on what feels expensive or problematic.

Re: Creepy Crawlies

#384

Post by mmooss »

akersten wrote: post #172
mmooss wrote: post #129

Earlier quoted context omitted.

Indeed. The solution isn't technical but legal. It's clearly abusive of - really stealing - other people's resources; there's no question about it. For some reason, like with fraud via email, text, and phone, we don't do anything about it. All this brazen crime and government does nothing; we don't even imagine government doing anything.

The brazen crime of asking for a resource and receiving it. You have the power, and should exercise it, to rate limit bad actors

> You have the power, and should exercise it, to rate limit bad actors

It's well known, and it was the point of the OP, that we don't have that power. If you know a solution, please share it - the world would love to hear it.

Re: Creepy Crawlies

#385

Post by leoqa »

inigyou wrote: post #295
okanat wrote: post #218

Earlier quoted context omitted.

What prevents people from obtaining or buying such devices and automating them? Using TVs as proxies is just one example of that. People will be willing to give their ID cards away too, if you pay them or beat them enough.

In fact they already do this. Buying 100 android phones and chargers is cheaper than reverse engineering whatever you're trying to automate - or was, before AI.

.. and we ban those device ids and move on. Your capital is lost.

Re: Creepy Crawlies

#386

Post by stjo »

semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience.

Lots of work has been done to do just that by monero’s proof of work algorithm. They achieved ASIC and GPU resistance by making you generate entire programs (with quite a bit of RAM usage) and finding one that produces a desired output. Then you submit the seed used for the program (a series of programs actually but let’s not get into the details)

Re: Creepy Crawlies

#388

Post by oowa »

kijin wrote: post #21
teo_zero wrote: post #10

Earlier quoted context omitted.

But there are "cubic bazillions" of possible URLs that are being requested. Even if they boil down to "only" some millions actual commits, their rendered HTMLs are all different.

Yes, this is very difficult to solve for sites with many URL variations, like git repos and heavily threaded forums. Your cache is always full, but the hit ratio is abysmal.

remember, its only difficult for you. at the end of the day its just 7gb of data...

Re: Creepy Crawlies

#389

Post by oowa »

TLDR basically old tech is not optimized for scrapers / slurpers / etc. to the point it would take 42^n to solve all possible combinations. Why? Murphy's law. Solution for OP is to ignore for now. Otherwise Use or invent something else. Easy enough. other notes... Anubis and other gatekeepers dont work perfectly, but ok for now.
Post reply on HN