Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

271–280 of 693 posts

Re: Creepy Crawlies

#271

Post by fer »

mattmcal wrote: post #181
afarah1 wrote: post #147

Why not aggressively rate limit? Legitimate use of HTML rendered commits should be largely unaffected, and crawlers slowed to a halt. You can even jail after a number of 429's...

There is a section in the article answering your question if you read it. > Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you.

If you operate your site alone in a silo yes, but crowdsourced solutions to ban/rate limit those IPs exist.

Re: Creepy Crawlies

#272

Post by marginalia_nu »

inigyou wrote: post #261
marginalia_nu wrote: post #99

Earlier quoted context omitted.

Yeah my search engine saw traffic of up to 160 queries per second the other day from some bot that was ostensibly searching for information on Jack Parsons. Just variations on the same query in different permutations of filters and site:-terms.

That would be a different bot, one written specifically for your site. Mainly we're discussing the dumb ones that just crawl all possible http links

No I think this is the same case. It seems to just be following local links on the SERP. From a search for jack parsons, you can find hyperlinks to the sorts of requests it's making.

Re: Creepy Crawlies

#273

Post by inigyou »

feelamee wrote: post #98
rwmj wrote: post #65

Earlier quoted context omitted.

It's a social problem, technical solutions aren't going to work. To solve social problems you need the law to get involved. It should simply be illegal to install malware on someone's TV and use it to hijack their internet connection, and the punishment should be the same as for any theft of service, including imprisonment for people who do it.

The law will not work without a technical solution. So, you need both. But, at least, technical.

You just take a sample of the crawling to whichever police department knows how to deal with DDoS, they subpoena the ISPs involved and trace who's using the proxies?

Re: Creepy Crawlies

#274

Post by inigyou »

ChocolateGod wrote: post #164

> phone gets uncomfortably warm as it's doing the number crunching IMHO if I visit your website and it intentionally starts wasting my electricity for no other reason than to cost me money, with no opt in, it's hostile and malicious.

You try hosting gitea in 2026. The only other option is taking the site offline.

Re: Creepy Crawlies

#275

Post by kgeist »

How about: "Type the seahorse emoji to solve the CAPTCHA" :) Something that triggers infinite loops in LLMs or trips the guardrails.

Re: Creepy Crawlies

#277

Post by oowa »

have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.

Re: Creepy Crawlies

#278

Post by inigyou »

Kamq wrote: post #191
Arubis wrote: post #165

Earlier quoted context omitted.

How do you define humanity? How do you ensure it includes every human? How do you ensure it doesn’t include every non-human? I’m not even asking about computation or algorithms. I straight up don’t think you can make a definition that isn’t a tautology or an approximation. Both of which are useful, but neither of which can fit a _proof_.

You're not wrong, but something can work well enough to still be useful despite falling short of the idea of a proof or any formal definition. Let's say that 95% of individual humans can pass it and only 2% of bots. For someone maintaining a website, who has to decide between using this system and shutting down their site because of the increased costs, that may very well be good enough

I'm one of the 5% apparently, cloudflare thinks I'm a bot. I suppose you'll exempt me in exchange for all my ID documents and bank statements?

Re: Creepy Crawlies

#280

Post by inigyou »

NooneAtAll3 wrote: post #60
tarpitt wrote: post #49

Maybe you could have a system that heuristicially detects when an crawler is making the request and then feeds them a modified page, itself generated from an LLM, that injects vulnerabilities and bad code and discussion and such.

it's hard to separate spambot that only accesses 3-5 links per IP and a legit user. Changing content for legit user can be devastating

Put a cookie wall in front. The bot will either load the cookie and have a persistent identifier, or not load the cookie and not get in
Post reply on HN