Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

181–190 of 680 posts

Re: Creepy Crawlies

#181

Post by mattmcal »

afarah1 wrote: post #147

Why not aggressively rate limit? Legitimate use of HTML rendered commits should be largely unaffected, and crawlers slowed to a halt. You can even jail after a number of 429's...

There is a section in the article answering your question if you read it.

> Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you.

Re: Creepy Crawlies

#182

Post by _ks3e »

lxgr wrote: post #127
NooneAtAll3 wrote: post #57

from what I see there are 2 solutions: 1) ban TV-proxy-as-a-service - straight up go to every representative there is and start pushing and lobbying and everything to stop spammers from distributing over non-computer devices, especially legally 2) make old commits more expensive to access than new ones. Legit users are not going to access those much, so they can pay the time. I assume diverse (unpredictable?) difficu…

> straight up go to every representative there is In every country of the world? I think you might have a significantly easier time convincing Samsung and LG to ban this via their T&C.

Good news: https://krebsonsecurity.com/2026/07/lg-to-ban-residential-pr...

Re: Creepy Crawlies

#184

Post by wongarsu »

Y_Y wrote: post #64
nneonneo wrote: post #59

Earlier quoted context omitted.

I disagree. The kernel finds it effective - 66% of scrapers are turned away directly. The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site. If each new site they hit forces…

The implication here is that the proxy fridge forwards the Anubis challenge to a dedicated rig controlled by the scraper who efficiently solves it and returns the answer.

That's still a notable step up in completely and resource investment for the crawler

See also how captchas continued being effective for years despite services like anti-captcha offering to solve them for you for a fifth of a cent each by farming the work out to India. It took advances in AI that made it viable to reliably solve them on-device to bring the end of the captcha

Re: Creepy Crawlies

#185

Post by oasisbob »

stickfigure wrote: post #167

How about just stop offering a html interface to the code? This doesn't seem like a critical service. Let people clone the repo normally. If someone else wants to run a public HTML service, let them deal with the bots. If you really want to offer a web interface, put it behind login. You can apply enough restrictions (captcha, super slow rate limit for new accounts) that it isn't cost effective to generate zillions o…

I think you underestimate the difficulty in effectively gating user registration. If the anti-bot efforts in general don't work for other site pages, they won't work for signup functionality either.

Re: Creepy Crawlies

#186

Post by mbirth »

Demiurge wrote: post #171
pdonis wrote: post #154

Earlier quoted context omitted.

> I don’t want to stop the bots from updating their knowledge based on all the content. Why not? At this point, any attempt to try to meet the bots halfway is doomed.

Because I also use AI, and I actually want the content from the site to show up for me, as well as other AI users. More over, I actually pay for three different AI subscriptions, and use them a lot. I also think it’s fair that I pay for their services. I just want them to also pay for what they use.

If these were actual legitimate bots using a known user-agent and coming from the same subnet so you could throttle their requests it would be fine.

But I’m seeing requests coming in from random IP addresses from all over the world using randomised user-agents. But somehow these are trying URLs in sequential order. A while ago this would be classed as a DDoS cyber attack.

Re: Creepy Crawlies

#187

Post by Aurornis »

wongarsu wrote: post #100
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them Fo…

> Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them

Depending on the configuration, Anubis will supply a token after the challenge that bypasses the challenge for a time.

So any scraper that retains basic cookies will be able to bypass the challenge for a number of page views.

A user who needs to load a single page and a bot that wants to scrape a number of pages may pay the same cost.

The amortized per-view cost is highest for the real user.

Re: Creepy Crawlies

#188

Post by pbronez »

“Expect to lose some functionality, at least when accessing our resources anonymously.”

This seems fine to me. It would be a better world if we could have anonymous bulk data access. But if aggressive scrapers are bloating host costs, I’m fine with logging in.

Now, the flip side is that ONCE logged in, I want my bulk access. The worst of all worlds with when you demand authentication and then STILL block bulk access.

Case in point, I want to automatically download my Amazon and Target order records. This is easy to automate with playwright or whatever, but authentication stays annoying. My sessions expire quickly and I have to re-auth all the time. There should be an API to pull this data down.

Re: Creepy Crawlies

#189

Post by persolb »

Demiurge wrote: post #26

I maintain a formerly popular gaming website, and it used to have hundreds of legitimate requests per second. The load would be especially high during popular event times. So, it’s always been running on a dedicated server. It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and d…

Maybe won’t work for your site; I’ve been leaving flat files open, putting anything that needs processing (like search and login) behind Anubis.

Re: Creepy Crawlies

#190

Post by Kuinox »

1.4 billions requests, 258 160 cpu hours. That's 1.5 requests per second ? I'm starting to believe, the issue is more that their software is not well optimized.
Post reply on HN