Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I block all 26M of your curl requests

foxmoss.com

Post reply on HN
Print view View original

21–30 of 74 posts

Re: How I block all 26M of your curl requests

#21

Post by coppsilgold »

There are also HTTP fingerprints. I believe it's named after akamai or something.

All of it is fairly easy to fake. JavaScript is the only thing that poses any challenge and what challenge it poses is in how you want to do it with minimal performance impact. The simple truth is that a motivated adversary can interrogate and match every single minor behavior of the browser to be bit-perfect and there is nothing anyone can do about it - except for TPM attestations which also require a full jailed OS environment in order to control the data flow to the TPM.

Even the attestation pathway can probably be defeated, either through the mandated(?) accessibility controls or going for more extreme measures. And putting the devices to work in a farm.

Re: How I block all 26M of your curl requests

#22

Post by jacquesm »

geocar wrote: post #19

Do you actually use this? $ md5 How\ I\ Block\ All\ 26\ Million\ Of\ Your\ Curl\ Requests.html MD5 (How I Block All 26 Million Of Your Curl Requests.html) = e114898baa410d15f0ff7f9f85cbcd9d (downloaded with Safari) $ curl https://foxmoss.com/blog/packet-filtering/ | md5sum e114898baa410d15f0ff7f9f85cbcd9d - I'm aware of curl-impersonate https://github.com/lwthiker/curl-impersonate which works around these kinds of th…

> Also also: 26M requests daily is only 300 requests per second and Apache could handle that easily over 15 years ago. Why worry about something as small as that?

That doesn't matter, does it? Those 26 million requests could be going to actual users instead and 300 requests per second is non-trivial if the requests require backend activity. Before you know it you're spending most of your infra money on keeping other people's bots alive.

Re: How I block all 26M of your curl requests

#25

Post by palmfacehn »

It is a cute technique, but I would prefer if the fingerprint were used higher up in the stack. The fingerprint should be compared against the User-Agent. I'm more interested in blocking curl when it is specifically reporting itself as Chrome/x.y.z.

Most of the abusive scraping is much lower hanging fruit. It is easy to identify the bots and relate that back to ASNs. You can then block all of Huawei cloud and the other usual suspects. Many networks aren't worth allowing at this point.

For the rest, the standard advice about performant sites applies.

Re: How I block all 26M of your curl requests

#26

Post by delusional »

coppsilgold wrote: post #21

There are also HTTP fingerprints. I believe it's named after akamai or something. All of it is fairly easy to fake. JavaScript is the only thing that poses any challenge and what challenge it poses is in how you want to do it with minimal performance impact. The simple truth is that a motivated adversary can interrogate and match every single minor behavior of the browser to be bit-perfect and there is nothing anyone…

This is exactly right, and it's why I believe we need to solve this problem in the human domain, with laws and accountability. We need new copyrights that cover serving content on the web, and gives authors control over who gets to access that content, WITHOUT requiring locked down operating systems or browser monopolies.

Re: How I block all 26M of your curl requests

#29

Post by arcfour »

jacquesm wrote: post #22
geocar wrote: post #19

Do you actually use this? $ md5 How\ I\ Block\ All\ 26\ Million\ Of\ Your\ Curl\ Requests.html MD5 (How I Block All 26 Million Of Your Curl Requests.html) = e114898baa410d15f0ff7f9f85cbcd9d (downloaded with Safari) $ curl https://foxmoss.com/blog/packet-filtering/ | md5sum e114898baa410d15f0ff7f9f85cbcd9d - I'm aware of curl-impersonate https://github.com/lwthiker/curl-impersonate which works around these kinds of th…

> Also also: 26M requests daily is only 300 requests per second and Apache could handle that easily over 15 years ago. Why worry about something as small as that? That doesn't matter, does it? Those 26 million requests could be going to actual users instead and 300 requests per second is non-trivial if the requests require backend activity. Before you know it you're spending most of your infra money on keeping other…

Blocking 26M bot requests doesn't mean 26M legitimate requests magically appear to take their place. The concern is that you're spending infrastructure resources serving requests that provide zero business value. Whether that matters depends on what those requests actually cost you. As the original commenter pointed out, this is likely not very much at all.

Re: How I block all 26M of your curl requests

#30

Post by baobun »

unwind wrote: post #4

I got exactly this far: uint8_t *data = (void *)(long)ctx->data; before I stopped reading. I had to go look up the struct xdp_md [1], it is declared like this: struct xdp_md { __u32 data; __u32 data_end; __u32 data_meta; /* ... further fields elided ... */ }; So clearly the `data` member is already an integer. The sane way to cast it would be to cast to the actual desired destination type, rather than first to some o…

Possibly stupid question: Why does the author use different types for data and data_end in their struct?
Post reply on HN