Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

701–706 of 706 posts

Re: Creepy Crawlies

#701

Post by iririririr »

Demiurge wrote: post #151
1oooqooq wrote: post #148

Earlier quoted context omitted.

See, this is you falling for their marketing. It makes sense doesn't it? If you own all the pipes, you can effectively block bad actors. But in reality what they do is to just block everyone who blocks advertising tracking and profiling tech (such as canvas fingerprint etc)... and because all vendors use cf, they all block those same clients, and they think there's some winner takes all magic happening with finely tu…

What would you suggest as an alternative? Block the bots yourself ?

don't write a crap site that fall to pieces with a million unauthenticated users *shrugemoji

Re: Creepy Crawlies

#702

Post by virgoerns »

pmlnr wrote: post #651
VladVladikoff wrote: post #357

Earlier quoted context omitted.

Have you tried blocking a million IPs before? Fail2ban gets pretty shaky at even 200,000 The AI crawler traffic I’ve seen sends one request per ip and seemingly has an infinite pool of residential IPs. You can’t block the ASNs becuase you also block honest clients. IP blocks are the wrong solution. And because I’m being negative I’ll also be constructive, IMHO the correct solution for fighting residential proxy crawl…

Fail2ban becomes a serious bottleneck at significant traffic. I've replaced it with a shell script and direct pf commands that run every few minutes.

Do you mind sharing your script? I also noticed that fail2ban currently eats ~25% of CPU and started thinking if it's even worth running at this point

Re: Creepy Crawlies

#703

Post by inigyou »

sgsjchs wrote: post #691
inigyou wrote: post #662

Earlier quoted context omitted.

Why else would a thing be programmed to drop cookies after every seconds request?

Maybe I misunderstand what you mean by "load the cookie". What is it if not Set-Cookie header with redirect status? That doesn't seem particularly special to me.

Set-Cookie header, 200 status and a JavaScript redirect? Many ways. Set-Cookie on pages that load quickly and check it on pages that load slowly?

Re: Creepy Crawlies

#704

Post by inigyou »

parineum wrote: post #695
inigyou wrote: post #279

Earlier quoted context omitted.

No they mean the people doing the actual work, I think.

The people doing the work aren't paying for AI, they're getting paid by AI. The people who are having the costs of users not paying "their fair share" (this phrase has officially jumped the shark) are the investors who are subsidizing these companies.

I'm getting paid for AI to scrape my stuff? That's news to me. How do I claim it?

Re: Creepy Crawlies

#705

Post by golem14 »

cobbzilla wrote: post #221

I ended public access to my git server after I got flooded by bots and my own commits were noticeably lagging. That’s not an option for the kernel. It’s hard to read the cat-and-mouse account with any hope today. I think the flood abates someday but not sure how it happens.

why not require an account for html access and otherwise just serve say a pre-cached complete repository requiring minimal work?

Re: Creepy Crawlies

#706

Post by imtringued »

semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

That's actually weaponizable. You can deliberately insist on an inefficient algorithm to find bots that use a better algorithm to bypass the PoW. Legitimate users have zero reason to redo the challenge over and over again, meanwhile bots want to constantly redo the challenge because they don't want you to throttle the bot based on the reused session cookie. You can then crank up the difficulty up over and over again.
Post reply on HN