I like this project!
Is there a way to request impersonization of the current version of Chrome (or whatever)?
Curl-Impersonate
21–30 of 114 posts
Re: Curl-Impersonate
#22The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyo…
A lot of the motivation comes from government regulations too. Right now this is mostly in banking, but social media and porn regs are coming too.
Re: Curl-Impersonate
#23What are some example sites where this is both necessary and sufficient? In my experience sites with serious anti-bot protection basically always have JavaScript-based browser detection, and some are capable of defeating puppeteer-extra-plugin-stealth even in headful mode. I doubt sites without serious anti-bot detection will do TLS fingerprinting. I guess it is useful for the narrower use case of getting a short-liv…
There are sites that will block curl and python-requests completely, but will allow curl-impersonate. IIRC, Amazon is an example that has some bot protection but it isn't "serious".
Re: Curl-Impersonate
#24What are some example sites where this is both necessary and sufficient? In my experience sites with serious anti-bot protection basically always have JavaScript-based browser detection, and some are capable of defeating puppeteer-extra-plugin-stealth even in headful mode. I doubt sites without serious anti-bot detection will do TLS fingerprinting. I guess it is useful for the narrower use case of getting a short-liv…
A lot of WAFs make it a simple thing to set up. Since it doesn't require any application-level changes, it's an easy "first move" in the anti-bot arms race.
At the time I wrote this up, r1-api.rabbit.tech required TLS client fingerprints to match an expected value, and not much else: https://gist.github.com/DavidBuchanan314/aafce6ba7fc49b19206...
(I haven't paid attention to what they've done since so it might no longer be the case)
Re: Curl-Impersonate
#25The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyo…
"I have nothing to hide" will eventually spread to everyone. Very unfortunate.
Re: Curl-Impersonate
#26The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyo…
> I can't help but feel like these are the dying breaths of the open Internet though I agree with the over zealous tracking by the megacorps but this is also due to bad actors, I work for a financial company and the amount of API abuse, ATO, DDoS, nefarious bot traffic, etc. we see on a daily basis is absolutely insane
But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up?
And when it does get more dangerous, is over zealous tracking the best counter for this?
I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to application firewalls and private subnets and whatnot. E.g. a large fai2ban rule to just ban anything that attempts to HTTP GET /admin.php or /phpmyadmin etc, even just once, gets rid of almost all nefarious bot traffic.
So, I think the amount of attacks indeed can be insane. But the amount that need over zealous tracking is to be countered, is, AFAICS, rather small.
Re: Curl-Impersonate
#27The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyo…
> I can't help but feel like these are the dying breaths of the open Internet though I agree with the over zealous tracking by the megacorps but this is also due to bad actors, I work for a financial company and the amount of API abuse, ATO, DDoS, nefarious bot traffic, etc. we see on a daily basis is absolutely insane
Yep totally agree these are problems. I don't have a good alternative proposal either, I'm just disappointed with what we're converging on.
Re: Curl-Impersonate
#28What are some example sites where this is both necessary and sufficient? In my experience sites with serious anti-bot protection basically always have JavaScript-based browser detection, and some are capable of defeating puppeteer-extra-plugin-stealth even in headful mode. I doubt sites without serious anti-bot detection will do TLS fingerprinting. I guess it is useful for the narrower use case of getting a short-liv…
A lot of WAFs make it a simple thing to set up. Since it doesn't require any application-level changes, it's an easy "first move" in the anti-bot arms race. At the time I wrote this up, r1-api.rabbit.tech required TLS client fingerprints to match an expected value, and not much else: https://gist.github.com/DavidBuchanan314/aafce6ba7fc49b19206... (I haven't paid attention to what they've done since so it might no lon…
Makes sense, thanks.
Re: Curl-Impersonate
#29Earlier quoted context omitted.
> I can't help but feel like these are the dying breaths of the open Internet though I agree with the over zealous tracking by the megacorps but this is also due to bad actors, I work for a financial company and the amount of API abuse, ATO, DDoS, nefarious bot traffic, etc. we see on a daily basis is absolutely insane
But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to appl…
> E.g. a large fai2ban rule to just ban anything that attempts to HTTP GET /admin.php or /phpmyadmin etc, even just once, gets rid of almost all nefarious bot traffic.
unfortunately fail2ban wouldn't even make a dent in the attack traffic hitting the endpoints in my day-to-day work, these are attackers utilizing residential proxy infrastructure that are increasingly capable of solving JS/client-puzzle challenges.. the arms race is always escalating
Re: Curl-Impersonate
#30The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyo…
"I have nothing to hide" will eventually spread to everyone. Very unfortunate.
I'm in a similar boat but it's more like "I have nothing I can hide".
These days I just tell friends & family to assume that nothing they do is private.