Creepy Crawlies • Hacker News Board

Re: Creepy Crawlies

#701

Post by iririririr » Tue, Sep 01, 2026, 5:37 AM UTC

Demiurge wrote: post #151

1oooqooq wrote: post #148

Earlier quoted context omitted.

See, this is you falling for their marketing. It makes sense doesn't it? If you own all the pipes, you can effectively block bad actors. But in reality what they do is to just block everyone who blocks advertising tracking and profiling tech (such as canvas fingerprint etc)... and because all vendors use cf, they all block those same clients, and they think there's some winner takes all magic happening with finely tu…

What would you suggest as an alternative? Block the bots yourself ?

don't write a crap site that fall to pieces with a million unauthenticated users *shrugemoji

Re: Creepy Crawlies

#702

Post by virgoerns » Tue, Sep 01, 2026, 5:49 AM UTC

pmlnr wrote: post #651

VladVladikoff wrote: post #357

Earlier quoted context omitted.

Have you tried blocking a million IPs before? Fail2ban gets pretty shaky at even 200,000 The AI crawler traffic I’ve seen sends one request per ip and seemingly has an infinite pool of residential IPs. You can’t block the ASNs becuase you also block honest clients. IP blocks are the wrong solution. And because I’m being negative I’ll also be constructive, IMHO the correct solution for fighting residential proxy crawl…

Fail2ban becomes a serious bottleneck at significant traffic. I've replaced it with a shell script and direct pf commands that run every few minutes.

Do you mind sharing your script? I also noticed that fail2ban currently eats ~25% of CPU and started thinking if it's even worth running at this point

Re: Creepy Crawlies

#703

Post by inigyou » Tue, Sep 01, 2026, 7:07 AM UTC

sgsjchs wrote: post #691

inigyou wrote: post #662

Earlier quoted context omitted.

Why else would a thing be programmed to drop cookies after every seconds request?

Maybe I misunderstand what you mean by "load the cookie". What is it if not Set-Cookie header with redirect status? That doesn't seem particularly special to me.

Set-Cookie header, 200 status and a JavaScript redirect? Many ways. Set-Cookie on pages that load quickly and check it on pages that load slowly?

Re: Creepy Crawlies

#704

Post by inigyou » Tue, Sep 01, 2026, 7:08 AM UTC

parineum wrote: post #695

inigyou wrote: post #279

Earlier quoted context omitted.

No they mean the people doing the actual work, I think.

The people doing the work aren't paying for AI, they're getting paid by AI. The people who are having the costs of users not paying "their fair share" (this phrase has officially jumped the shark) are the investors who are subsidizing these companies.

I'm getting paid for AI to scrape my stuff? That's news to me. How do I claim it?

Re: Creepy Crawlies

#705

Post by golem14 » Tue, Sep 01, 2026, 7:40 AM UTC

cobbzilla wrote: post #221

I ended public access to my git server after I got flooded by bots and my own commits were noticeably lagging. That’s not an option for the kernel. It’s hard to read the cat-and-mouse account with any hope today. I think the flood abates someday but not sure how it happens.

why not require an account for html access and otherwise just serve say a pre-cached complete repository requiring minimal work?