Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I block all 26M of your curl requests

foxmoss.com

Post reply on HN
Print view View original

51–60 of 74 posts

Re: How I block all 26M of your curl requests

#51

Post by ospider »

mrb wrote: post #38

" There’s ways to get around TLS signatures but it’s much harder and requires a lot more legwork to get working " I wouldn't call it "much harder". All you need to bypass the signature is to choose random ciphers (list at https://curl.se/docs/ssl-ciphers.html ) and you mash them up in a random order separated by colons in curl's --ciphers option. If you pick 15 different ciphers in a random order, there are over a tr…

It can be much more easier and realistic with https://github.com/lexiforest/curl-impersonate.

Re: How I block all 26M of your curl requests

#52

Post by noAnswer »

dancek wrote: post #32
geocar wrote: post #19

Do you actually use this? $ md5 How\ I\ Block\ All\ 26\ Million\ Of\ Your\ Curl\ Requests.html MD5 (How I Block All 26 Million Of Your Curl Requests.html) = e114898baa410d15f0ff7f9f85cbcd9d (downloaded with Safari) $ curl https://foxmoss.com/blog/packet-filtering/ | md5sum e114898baa410d15f0ff7f9f85cbcd9d - I'm aware of curl-impersonate https://github.com/lwthiker/curl-impersonate which works around these kinds of th…

The article talks about 26M requests per second . It's theoretical, of course.

Not requests, packets: "And according to some benchmarks Wikipedia cites, you can drop 26 million packets per second on consumer hardware."

The Number in the Title is basically fantasy. (Not based on the authors RL experience.) So is saying a DDoS is well distributed over 24 hours.

Re: How I block all 26M of your curl requests

#53

Post by dpoloncsak »

delusional wrote: post #26
coppsilgold wrote: post #21

There are also HTTP fingerprints. I believe it's named after akamai or something. All of it is fairly easy to fake. JavaScript is the only thing that poses any challenge and what challenge it poses is in how you want to do it with minimal performance impact. The simple truth is that a motivated adversary can interrogate and match every single minor behavior of the browser to be bit-perfect and there is nothing anyone…

This is exactly right, and it's why I believe we need to solve this problem in the human domain, with laws and accountability. We need new copyrights that cover serving content on the web, and gives authors control over who gets to access that content, WITHOUT requiring locked down operating systems or browser monopolies.

>with laws and accountability.

Isn't this how we get EU's digital ID nonsense? Otherwise, how do you hold an anon user behind 5 proxies accountable? What if its from a foreign country?

Re: How I block all 26M of your curl requests

#54

Post by chuckadams »

stingraycharles wrote: post #10
seba_dos1 wrote: post #8

Earlier quoted context omitted.

The problem you usually attempt to alleviate by using Anubis is that you get hit by load generated by aggressive AI scrappers that are otherwise indistinguishable from real users. As soon as the bot is polite enough to identify as some kind of a bot, the problem's gone, as you can apply your regular measures for rate limiting and access control now. (yes, there are also people who use it as an anti-AI statement, but…

Yeah that makes sense. Bad players will try to look like a regular browser, good players will have no problems revealing they’re a bot.

Very similar to when I did anti-spam: the low-end spamware would make shoddy attempts at forging MUA headers that ended up being red flags that made for cheap detection rules. The successful spamware omitted extraneous headers altogether and was more standards-compliant than many of the actually legit MUAs.

Re: How I block all 26M of your curl requests

#55

Post by klaussilveira »

geocar wrote: post #19

Do you actually use this? $ md5 How\ I\ Block\ All\ 26\ Million\ Of\ Your\ Curl\ Requests.html MD5 (How I Block All 26 Million Of Your Curl Requests.html) = e114898baa410d15f0ff7f9f85cbcd9d (downloaded with Safari) $ curl https://foxmoss.com/blog/packet-filtering/ | md5sum e114898baa410d15f0ff7f9f85cbcd9d - I'm aware of curl-impersonate https://github.com/lwthiker/curl-impersonate which works around these kinds of th…

> so there's literally no evidence of mechanical means.

Keystroke dynamics and mouse movement analysis are pretty fun ways to tackle more advanced bots: https://research.roundtable.ai/proof-of-human/

But of course, it is a game of cat and mouse and there are ways to simulate it.

Re: How I block all 26M of your curl requests

#56

Post by efilife »

klaussilveira wrote: post #55
geocar wrote: post #19

Do you actually use this? $ md5 How\ I\ Block\ All\ 26\ Million\ Of\ Your\ Curl\ Requests.html MD5 (How I Block All 26 Million Of Your Curl Requests.html) = e114898baa410d15f0ff7f9f85cbcd9d (downloaded with Safari) $ curl https://foxmoss.com/blog/packet-filtering/ | md5sum e114898baa410d15f0ff7f9f85cbcd9d - I'm aware of curl-impersonate https://github.com/lwthiker/curl-impersonate which works around these kinds of th…

> so there's literally no evidence of mechanical means. Keystroke dynamics and mouse movement analysis are pretty fun ways to tackle more advanced bots: https://research.roundtable.ai/proof-of-human/ But of course, it is a game of cat and mouse and there are ways to simulate it.

I don't think that mouse movement analysis is used anywhere. But it was reportedly used 10 years ago by Google's captcha. This is a client side check than can trivially be bypassed

Re: How I block all 26M of your curl requests

#58

Post by peetistaken »

coppsilgold wrote: post #21

There are also HTTP fingerprints. I believe it's named after akamai or something. All of it is fairly easy to fake. JavaScript is the only thing that poses any challenge and what challenge it poses is in how you want to do it with minimal performance impact. The simple truth is that a motivated adversary can interrogate and match every single minor behavior of the browser to be bit-perfect and there is nothing anyone…

Indeed, I named it after akamai because they wrote a whitepaper for it. I think I first used akamai_fingerprint on https://tls.peet.ws, where you can see all your fingerprints!

Re: How I block all 26M of your curl requests

#59

Post by vanyle »

geek_at wrote: post #34

btw you opensourced also your website ~$ curl https://foxmoss.com/.git/config [core] repositoryformatversion = 0 filemode = true bare = false logallrefupdates = true [remote "origin"] url = https://github.com/FoxMoss/PersonalWebsite fetch = +refs/heads/ :refs/remotes/origin/ [branch "master"] remote = origin merge = refs/heads/master

The git seems to only contain the build of the website with no source code.

The author is probably using git to push the content to the hosting server as an rsync alternative, but there does not seem to be much leaked information, apart from the url of the private repository.

Re: How I block all 26M of your curl requests

#60

Post by piggg »

mrweasel wrote: post #40
piggg wrote: post #31

Blocking on ja3/ja4 signals to folks exactly what you are up to. This is why bad actors doing ja3 randomization became a thing in the last few years and made ja3 matching useless. Imo use ja3/ja4 as a signal and block on src IP. Don't show your cards. Ja4 extensions that use network vs http/tls latency is also pretty elite to identify folks proxying.

Some of the bad actors, and Chrome, randomize extensions, but only their order. I think it's ja3n that started to sort the extensions, before doing the hashing. Blocking on source IP is tricky, because that frequently means blocking or rate-limiting thousands of IPs. If you're fine with just blocking entire subnets or all of AWS, I'd agree that it's probably better. It really depends on who your audience is and who t…

For my use cases I block src IP for some period of time (minutes). I don't block large pools of IPs as the blast radius is too large. That said - there are well established shit hosters who provide multiple /24s to proxy/dirty VPN types that are generally bad.
Post reply on HN