Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Curl-Impersonate

github.com

Post reply on HN
Print view View original

91–100 of 114 posts

Re: Curl-Impersonate

#91

Post by int0x29 »

fijiaarone wrote: post #81
mattpallissard wrote: post #40

Earlier quoted context omitted.

That's not the same type of bot net. Fail 2 ban simply is not going to work when you have a popular unauthenticated endpoint. You have hundreds of thousands of rps spread across thousands of legitimate networks that. The requests are always modified to look legitimate in a never ending game of whack-a-mole. You wind up having to use things like tls fingerprinting with other heuristics to identify what to traffic to r…

Easy solution to rate limit. Require initial request to get 1 time token with a 1 second delay And then require valid requests to include the token. The token returned has a salt with something like timestamp and ip. That way they can only bombard the token generator. get /token Returns token with timestamp in salted hash get /resource?token=abc123xyz Check for valid token and drop or deny.

As at least one person working on this has pointed out in this thread: their adversaries have IP blocks and ASNs.

Re: Curl-Impersonate

#92

Post by crtasm »

userbinator wrote: post #84

I can't help but think that projects like these shouldn't be posted here, since the enemy is among us. Prodding the bear even more might lead to an acceleration towards the dystopia that others here have already prophesised. The following browsers can be impersonated. ...unfortunately no Firefox to be seen. I've had to fight this too, since I use a filtering proxy. User-agent discrimination should be illegal. One may…

It says "Firefox(In progress)", and the original project this was forked from has it: https://github.com/lwthiker/curl-impersonate

Re: Curl-Impersonate

#93

Post by dwattttt »

userbinator wrote: post #86
cle wrote: post #16

The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyo…

They've been planning this stuff for a long time... https://en.wikipedia.org/wiki/Next-Generation_Secure_Computi... ...and we're seeing the puzzle pieces fall into place. Mandated driver signing, TPMs, and more recently remote attestation. "Security" has always been the excuse --- securing their control over you.

Another trending thread right now is Pegasus/Predator; as much as it may be a facade, to say MS (or any OS vendor) has no business working on security/secure computing is demonstrably false.

Re: Curl-Impersonate

#94

Post by Szpadel »

shaky-carrousel wrote: post #80
Szpadel wrote: post #54

Earlier quoted context omitted.

We had captcha, but it was at later stage of the checkout process. This API endpoint needed to work from cached pages, so it could not contain any dynamic state in request. Some bots checked product page where we had info if product is in stock (although they tried heavenly to bypass any caches by putting garbage in URL). This kind of bots also scaled instantly to thousands checkout requests when product become avail…

Hm, is probably too late, but you could have implemented in your API calls some kind of proof of work. Something that's not too onerous for a casual user but it is hard for someone trying multiple requests.

This was actually one of my ideas how to solve it, observed behaviour strongly suggested that all those thousands of IP addresses where used by single server. Even small PoW with this volume should heavly influence their capacity. But we decided that we did not want to affect performance of mobile users. We later learned that such strategy is also used by cloudflare js check

Re: Curl-Impersonate

#95

Post by matheusmoreira »

cle wrote: post #16

The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyo…

You are on point. There is no open internet without computing freedom.

Computers used to be empowering. Cryptography used to be empowering. Then these corporations started using both against us. They own the computers now. Hardware cryptography ensures the computers only run their software now, software that does their the corporation's bidding and enforces their controls. And if we somehow gain control of the computer we are denied every service and essentially ostracized. I don't think it will be long before we are banned from the internet proper for using "unauthorized" devices.

It's an incredibly depressing state of affairs. Everything the word "hacker" ever stood for is pretty much dying. It feels like there's no way out.

Re: Curl-Impersonate

#96

Post by sangnoir »

zzo38computer wrote: post #78
sangnoir wrote: post #72

Earlier quoted context omitted.

Speaking as a person who has played on both offense and defense: this is a heuristic that's not used frequently enough by defenders. Clients that load a single HTML/JSON endpoint without loading css or image resources associated with the endpoints are likely bots (or user agents with a fully loaded cache, but defenders control what gets cached by legit clients and how). Bot data thriftiness is a huge signal.

Even legitimate users might want to disable CSS and pictures and whatever, and I often do when I just want to read the document. Blind users also might have no use for the pictures, and another possibility is if the document is longer than the screen so the picture is out of view then the user might program the client software to use lazy loading, etc.

Indeed, that's why it's one heuristic/signal among many others

Re: Curl-Impersonate

#97

Post by RiverCrochet »

josephcsible wrote: post #69
throwaway99210 wrote: post #29

Earlier quoted context omitted.

> E.g. a large fai2ban rule to just ban anything that attempts to HTTP GET /admin.php or /phpmyadmin etc, even just once, gets rid of almost all nefarious bot traffic. unfortunately fail2ban wouldn't even make a dent in the attack traffic hitting the endpoints in my day-to-day work, these are attackers utilizing residential proxy infrastructure that are increasingly capable of solving JS/client-puzzle challenges.. th…

Such a rule is a great way to let malicious users lock out a bunch of your legitimate customers. Imagine if someone makes a forum post and includes this in it: [img]https://example.com/phpmyadmin/whatever.png[/img]

That would be in the body of the request. OP is talking about URLs in the actual request, which is part of the header.

While I don't have experience with a great number of WAFs I'm sure sophisticated ones let you be quite specific on where you are matching text to identify bad requests.

As an aside, another "easy win" is assuming any incoming HTTP request for a dotfile is malicious. I see constant unsolicitied attempts to access `.env`, for example.

Re: Curl-Impersonate

#98

Post by josephcsible »

RiverCrochet wrote: post #97
josephcsible wrote: post #69

Earlier quoted context omitted.

Such a rule is a great way to let malicious users lock out a bunch of your legitimate customers. Imagine if someone makes a forum post and includes this in it: [img]https://example.com/phpmyadmin/whatever.png[/img]

That would be in the body of the request. OP is talking about URLs in the actual request, which is part of the header. While I don't have experience with a great number of WAFs I'm sure sophisticated ones let you be quite specific on where you are matching text to identify bad requests. As an aside, another "easy win" is assuming any incoming HTTP request for a dotfile is malicious. I see constant unsolicitied attemp…

When legitimate users viewed that forum post, their browsers would, in the course of loading the image, attempt to HTTP GET /phpmyadmin/whatever.png, with that being the URL in the actual request in the header.

Re: Curl-Impersonate

#99

Post by jillyboel »

Szpadel wrote: post #36
berkes wrote: post #26

Earlier quoted context omitted.

But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to appl…

I can tell you about my experience with blocking traffic from scalpers bots that were very active during pandemic. All requests produced by those bots were valid ones, nothing that could be flagged by tools like fail2ban etc (my assumption is that it would be the same for financial systems). Any blocking or rate limiting by IP is useless, we saw about 2-3 requests per minute per IP, and those actors had access to rid…

The best solution is to put everyone in a little cage and point and keep a permanent record of everything they do. This doesn't mean it's a desirable solution.

Re: Curl-Impersonate

#100

Post by jsnell »

fijiaarone wrote: post #82
jsnell wrote: post #42

Earlier quoted context omitted.

The question is a bit of a non sequitur, since this is not tracking. The TLS fingerprint is not a useful tracking vector, by itself nor as part of some composite fingerprint.

The point is that you have to use an approved client (eg browser, os) with an approved cert authority that goes through approved gatekeepers (eg Cloudflare, Akamai)

That seems pretty unlikely to be the original point of https://news.ycombinator.com/item?id=42549415, which mentions none of that, and doesn't even have directionally the same concerns.

But also, what you wrote is basically nonsense. Clients don't need "an approved cert authority". Nor are there any "approved gatekeepers", all major browsers are equally happy connecting to your Raspberry Pi as they are connecting to Cloudflare.

Post reply on HN