Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

701–710 of 715 posts

Re: Creepy Crawlies

#701

Post by iririririr »

Demiurge wrote: post #151
1oooqooq wrote: post #148

Earlier quoted context omitted.

See, this is you falling for their marketing. It makes sense doesn't it? If you own all the pipes, you can effectively block bad actors. But in reality what they do is to just block everyone who blocks advertising tracking and profiling tech (such as canvas fingerprint etc)... and because all vendors use cf, they all block those same clients, and they think there's some winner takes all magic happening with finely tu…

What would you suggest as an alternative? Block the bots yourself ?

don't write a crap site that fall to pieces with a million unauthenticated users *shrugemoji

Re: Creepy Crawlies

#702

Post by virgoerns »

pmlnr wrote: post #651
VladVladikoff wrote: post #357

Earlier quoted context omitted.

Have you tried blocking a million IPs before? Fail2ban gets pretty shaky at even 200,000 The AI crawler traffic I’ve seen sends one request per ip and seemingly has an infinite pool of residential IPs. You can’t block the ASNs becuase you also block honest clients. IP blocks are the wrong solution. And because I’m being negative I’ll also be constructive, IMHO the correct solution for fighting residential proxy crawl…

Fail2ban becomes a serious bottleneck at significant traffic. I've replaced it with a shell script and direct pf commands that run every few minutes.

Do you mind sharing your script? I also noticed that fail2ban currently eats ~25% of CPU and started thinking if it's even worth running at this point

Re: Creepy Crawlies

#703

Post by inigyou »

sgsjchs wrote: post #691
inigyou wrote: post #662

Earlier quoted context omitted.

Why else would a thing be programmed to drop cookies after every seconds request?

Maybe I misunderstand what you mean by "load the cookie". What is it if not Set-Cookie header with redirect status? That doesn't seem particularly special to me.

Set-Cookie header, 200 status and a JavaScript redirect? Many ways. Set-Cookie on pages that load quickly and check it on pages that load slowly?

Re: Creepy Crawlies

#704

Post by inigyou »

parineum wrote: post #695
inigyou wrote: post #279

Earlier quoted context omitted.

No they mean the people doing the actual work, I think.

The people doing the work aren't paying for AI, they're getting paid by AI. The people who are having the costs of users not paying "their fair share" (this phrase has officially jumped the shark) are the investors who are subsidizing these companies.

I'm getting paid for AI to scrape my stuff? That's news to me. How do I claim it?

Re: Creepy Crawlies

#705

Post by golem14 »

cobbzilla wrote: post #221

I ended public access to my git server after I got flooded by bots and my own commits were noticeably lagging. That’s not an option for the kernel. It’s hard to read the cat-and-mouse account with any hope today. I think the flood abates someday but not sure how it happens.

why not require an account for html access and otherwise just serve say a pre-cached complete repository requiring minimal work?

Re: Creepy Crawlies

#706

Post by imtringued »

semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

That's actually weaponizable. You can deliberately insist on an inefficient algorithm to find bots that use a better algorithm to bypass the PoW. Legitimate users have zero reason to redo the challenge over and over again, meanwhile bots want to constantly redo the challenge because they don't want you to throttle the bot based on the reused session cookie. You can then crank up the difficulty up over and over again.

Re: Creepy Crawlies

#707

Post by Lerc »

desterothx wrote: post #643
Lerc wrote: post #82

Earlier quoted context omitted.

As the article states, this phenomenon may be documented, but there is no consensus that it describes any practical reality. The predicted consequences have now had time to manifest, and have not done so. This makes the claim either false or overstated. Perhaps there will be issues in the future, but to date there have been many claims that AI development will stall (for a variety of reasons). If they were the critic…

You state it as if its only the quality of the hypothesis that matters, but you are ignoring an important part of it, timing. During the 08 financial crisis Burry had a hypothesis that was correct, however he almost went bankrupt still because he thought it would happen earlier than it did because of the government bailouts. He was pushing the day forward, and was looking like a "hapless doomsday cult". His hypothesi…

If you can't say when something will happen you are just playing with statistics on another domain.

Seems like this chaps problem was that the actions were based upon a different hypothesis than the one he stated. X will cause Y by the end of the year is considerably different to X will eventually cause Y.

Re: Creepy Crawlies

#708

Post by latentsea »

Dylan16807 wrote: post #506
latentsea wrote: post #445

Earlier quoted context omitted.

It's kinda interesting to see simultaneously the 'holy shit' response to the OpenAI / HuggingFace incident, and then the griping about Fable's controls regarding this.

"It should write the code I tell it to in an interactive session. Also when running autonomously it shouldn't decide to hack into systems." I don't see much connection between that problem and these controls.

What's the difference when you prompt it versus when it prompts itself?

Re: Creepy Crawlies

#709

Post by pmlnr »

virgoerns wrote: post #702
pmlnr wrote: post #651

Earlier quoted context omitted.

Fail2ban becomes a serious bottleneck at significant traffic. I've replaced it with a shell script and direct pf commands that run every few minutes.

Do you mind sharing your script? I also noticed that fail2ban currently eats ~25% of CPU and started thinking if it's even worth running at this point

https://petermolnar.net/article/asn-blocking-freebsd-pf-ngin...

It's not the most sophisticated thing.

Re: Creepy Crawlies

#710

Post by cobbzilla »

golem14 wrote: post #705
cobbzilla wrote: post #221

I ended public access to my git server after I got flooded by bots and my own commits were noticeably lagging. That’s not an option for the kernel. It’s hard to read the cat-and-mouse account with any hope today. I think the flood abates someday but not sure how it happens.

why not require an account for html access and otherwise just serve say a pre-cached complete repository requiring minimal work?

Because I’m lazy. It has HTTP basic auth. If anyone wants access they can message me directly.
Post reply on HN