Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Curl-Impersonate

github.com

Post reply on HN
Print view View original

81–90 of 114 posts

Re: Curl-Impersonate

#81

Post by fijiaarone »

mattpallissard wrote: post #40
berkes wrote: post #26

Earlier quoted context omitted.

But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to appl…

That's not the same type of bot net. Fail 2 ban simply is not going to work when you have a popular unauthenticated endpoint. You have hundreds of thousands of rps spread across thousands of legitimate networks that. The requests are always modified to look legitimate in a never ending game of whack-a-mole. You wind up having to use things like tls fingerprinting with other heuristics to identify what to traffic to r…

Easy solution to rate limit. Require initial request to get 1 time token with a 1 second delay And then require valid requests to include the token. The token returned has a salt with something like timestamp and ip. That way they can only bombard the token generator.

get /token

Returns token with timestamp in salted hash

get /resource?token=abc123xyz

Check for valid token and drop or deny.

Re: Curl-Impersonate

#82

Post by fijiaarone »

jsnell wrote: post #42
berkes wrote: post #26

Earlier quoted context omitted.

But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to appl…

The question is a bit of a non sequitur, since this is not tracking. The TLS fingerprint is not a useful tracking vector, by itself nor as part of some composite fingerprint.

The point is that you have to use an approved client (eg browser, os) with an approved cert authority that goes through approved gatekeepers (eg Cloudflare, Akamai)

Re: Curl-Impersonate

#83

Post by thatcat »

miki123211 wrote: post #65
bornfreddy wrote: post #48

Earlier quoted context omitted.

> Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements. That doesn't compute... Captcha is almost always used in such setups. It also looks like you could just offer an API endpoint which would return if the article is in stock or not, or even provide a webhook. Why fight them? Just make the resource usage lighter. I'm curious now though wha…

> Why fight them? Just make the resource usage lighter. Because you presumably want real, returning customers, and that means those customers need to get a chance at buying those products, instead of them being scooped up by a scalper the millisecond they appear on the website.

maybe offer the item for preorder ?

Re: Curl-Impersonate

#84

Post by userbinator »

I can't help but think that projects like these shouldn't be posted here, since the enemy is among us. Prodding the bear even more might lead to an acceleration towards the dystopia that others here have already prophesised.

The following browsers can be impersonated.

...unfortunately no Firefox to be seen.

I've had to fight this too, since I use a filtering proxy. User-agent discrimination should be illegal. One may think the EU could have some power to change things, but then again, they're also hugely into the whole "digital identity" thing.

Re: Curl-Impersonate

#85

Post by jsdwarf »

Szpadel wrote: post #47
stareatgoats wrote: post #43

Earlier quoted context omitted.

Blocking scalper bot traffic by any means, be it by source or certified identification seems a lost cause, i.e. not possible because it can always be circumvented. Why did you not have that filter at point of sale instead? I'm sure there are reasons, but to have a battery of captchas and a limit on purchases per credit card seems on the surface much more sturdy. And it doesn't require that everyone browsing the inter…

The product they tried to buy what not in stock anyways, but their strategy was to constantly try anyways, so in case it would become in stock they would be the first to get it. It was all for guest checkout, so no address yet to validate nor credit card. Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements. As stated before the main reason…

Disabling guest checkout would have been my weapon of choice or at least requiring the user to enter an email address to so that they are notified when the product becomes available.

Re: Curl-Impersonate

#86

Post by userbinator »

cle wrote: post #16

The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyo…

They've been planning this stuff for a long time...

https://en.wikipedia.org/wiki/Next-Generation_Secure_Computi...

...and we're seeing the puzzle pieces fall into place. Mandated driver signing, TPMs, and more recently remote attestation. "Security" has always been the excuse --- securing their control over you.

Re: Curl-Impersonate

#87

Post by userbinator »

jandrese wrote: post #56

The build scripts on this repo seem a bit cursed. It uses autotools but has you build them in a subdirectory. The default built target is a help text instead of just building the project. When you do use the listed build target it doesn't have the dependencies set up correctly so you have to run it like 6 times to get to the point where it is building the application. Ultimately I was not able to get it to build beca…

Look on the bright side: the harder it is to build and use correctly, the harder it is for the enemy to analyse and react.

Re: Curl-Impersonate

#88

Post by jagged-chisel »

cle wrote: post #16

The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyo…

> … helpful for making HTTP reqs without the overhead of running an entire browser stack

For those less informed, add “to impersonate the fingerprints of a browser.”

One can, obviously, make requests without a browser stack.

Re: Curl-Impersonate

#89

Post by dspillett »

Szpadel wrote: post #47
stareatgoats wrote: post #43

Earlier quoted context omitted.

Blocking scalper bot traffic by any means, be it by source or certified identification seems a lost cause, i.e. not possible because it can always be circumvented. Why did you not have that filter at point of sale instead? I'm sure there are reasons, but to have a battery of captchas and a limit on purchases per credit card seems on the surface much more sturdy. And it doesn't require that everyone browsing the inter…

The product they tried to buy what not in stock anyways, but their strategy was to constantly try anyways, so in case it would become in stock they would be the first to get it. It was all for guest checkout, so no address yet to validate nor credit card. Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements. As stated before the main reason…

> Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements

A time sensitive hash validating each request makes it a bit harder for them without significant extra work on your part. Address sensitive is much more effective but can result in issues for users that switch between networks (using your site on the move and passing between workers networks, for instance).

Re: Curl-Impersonate

#90

Post by dspillett »

geysersam wrote: post #76
miki123211 wrote: post #65

Earlier quoted context omitted.

> Why fight them? Just make the resource usage lighter. Because you presumably want real, returning customers, and that means those customers need to get a chance at buying those products, instead of them being scooped up by a scalper the millisecond they appear on the website.

Sounds like a dream having customers scooping up your products the millisecond the appear on the website. They should increase their prices.

No matter what the price, they would still have the "As stated before the main reason we needed to block it was volume of the traffic" problem that was stated above, for a popular item. In fact increasing the base price might attract even more scalpers and such.
Post reply on HN