Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

171–180 of 686 posts

Re: Creepy Crawlies

#171

Post by Demiurge »

pdonis wrote: post #154
Demiurge wrote: post #26

I maintain a formerly popular gaming website, and it used to have hundreds of legitimate requests per second. The load would be especially high during popular event times. So, it’s always been running on a dedicated server. It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and d…

> I don’t want to stop the bots from updating their knowledge based on all the content. Why not? At this point, any attempt to try to meet the bots halfway is doomed.

Because I also use AI, and I actually want the content from the site to show up for me, as well as other AI users.

More over, I actually pay for three different AI subscriptions, and use them a lot. I also think it’s fair that I pay for their services. I just want them to also pay for what they use.

Re: Creepy Crawlies

#172

Post by akersten »

mmooss wrote: post #129
singpolyma3 wrote: post #76

Why is no one filing lawsuits over this yet?

Indeed. The solution isn't technical but legal. It's clearly abusive of - really stealing - other people's resources; there's no question about it. For some reason, like with fraud via email, text, and phone, we don't do anything about it. All this brazen crime and government does nothing; we don't even imagine government doing anything.

The brazen crime of asking for a resource and receiving it.

You have the power, and should exercise it, to rate limit bad actors

Re: Creepy Crawlies

#173

Post by voakbasda »

alwa wrote: post #159
voakbasda wrote: post #71

Earlier quoted context omitted.

I would think that residential proxying would be illegal already, as it is a network intrusion. The trick is chasing down the offenders, proving their actions did harm, and getting them to pay. None of those steps are easy, even if there are laws to assist. Otherwise, spam would be a solved problem.

Is it still an intrusion if the user accepted the shrinkwrap TOS of an app that trades them “free TV” in exchange for allowing that app to operate a proxy (via an “app monetization” SDK) on their network?

Yeah, no one actually agrees to all of the individual terms in EULAs. That’s the first sign that the law will be nearly useless to address any aspect of these problems. It is already one-sided, and that side is not a friend to the consumer or general public.

Re: Creepy Crawlies

#174

Post by nicman23 »

dunder_cat wrote: post #24
nicman23 wrote: post #16

couldn't you have anubis on a dynamic difficulty? ie if a ip requests more than 1k pages per day +1 the difficulty ?

Yes, but the article (not to call you out - I just think it's a very important point!) points out that this type of throttling would not be effective: > Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the tim…

yeah i my head i thought they meant 4-5 _K_ requests

Re: Creepy Crawlies

#176

Post by iamniels »

jopsen wrote: post #139

I've seen this too. I think it's a few bad actors really. Because nobody serious about indexing content will do what these crawlers are doing.. They are consume lots of content that is unoriginal or duplicate or duplicate with minor modifications. Not sure how to block, but maybe a little bit of law enforcement could dramatically reduce the number of TVs being used a proxies.

I run a website with 10k unique pages. If I leave the gates open, Meta hits it 200.000 times per day. Every day. What are you paying developers $500k for Mark?

Re: Creepy Crawlies

#177

Post by jopsen »

robotmay wrote: post #169

I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…

Probably they must be deduplicating text they've seen before.

The only punishment would be unique text that trains their models to be degenerate. And even then you'd probably have to serve across many domains.

Re: Creepy Crawlies

#178

Post by jwilk »

radlad wrote: post #132
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

I'm reminded of LUKS Argon2 implementation, which is much slower than userspace. Therefore a sufficiently difficult challenge that may take less than a second in userspace Linux can take 20 or more seconds at boot.

Huh? Are you saying there's an Argon2 implementation in the kernel?

Re: Creepy Crawlies

#179

Post by embedding-shape »

robotmay wrote: post #169

I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…

> I have an admin dashboard with a little leaderboard for which ones get the most stuffed

You can't tease us like that without showing screenshots.

Re: Creepy Crawlies

#180

Post by oasisbob »

nneonneo wrote: post #59
tptacek wrote: post #44

Tavis Ormandy called this, about Anubis, almost exactly a year ago: https://news.ycombinator.com/item?id=44962529 It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper…

I disagree. The kernel finds it effective - 66% of scrapers are turned away directly. The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site. If each new site they hit forces…

> Many of these boxes may not even have the spare CPU power ...

I don't think that's generally how these networks use residential exit proxies. There are at least a dozen well-developed frameworks out there for decoupling the crawler from the network exit point.

Most res proxy exits are just slinging bytes for clients using SOCKS, or another tunneling protocol.

If nothing else, a modern scraper will want better control over their TLS fingerprints, and you can't get that if you're depending on the on-device TLS libraries alone.

Post reply on HN