Thankfully only a small fraction of website does JA3/JA4 fingerprinting. Some do more advanced stuff like correlating headers to the fingerprint. We have been able to get away without doing much in Caido for a long time but I am working on an OSS rust based equivalent. Neat trick, you can use the fingerprint of our competitor (Burp Suite) since it is whitelisted for the security folks to do their job. Only time you will not hear me complain about checkbox security.
Curl-Impersonate
41–50 of 114 posts
Re: Curl-Impersonate
#42Earlier quoted context omitted.
> I can't help but feel like these are the dying breaths of the open Internet though I agree with the over zealous tracking by the megacorps but this is also due to bad actors, I work for a financial company and the amount of API abuse, ATO, DDoS, nefarious bot traffic, etc. we see on a daily basis is absolutely insane
But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to appl…
The question is a bit of a non sequitur, since this is not tracking. The TLS fingerprint is not a useful tracking vector, by itself nor as part of some composite fingerprint.
Re: Curl-Impersonate
#43Earlier quoted context omitted.
But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to appl…
I can tell you about my experience with blocking traffic from scalpers bots that were very active during pandemic. All requests produced by those bots were valid ones, nothing that could be flagged by tools like fail2ban etc (my assumption is that it would be the same for financial systems). Any blocking or rate limiting by IP is useless, we saw about 2-3 requests per minute per IP, and those actors had access to rid…
Blocking scalper bot traffic by any means, be it by source or certified identification seems a lost cause, i.e. not possible because it can always be circumvented. Why did you not have that filter at point of sale instead? I'm sure there are reasons, but to have a battery of captchas and a limit on purchases per credit card seems on the surface much more sturdy. And it doesn't require that everyone browsing the internet announce their full name and residential address in order to satisfy the requirements of a social score ...
Re: Curl-Impersonate
#44The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyo…
Even if the internet was wide open it’s of little use these days.
AI will replace any search you would want to do to find information, the only reason to scour the internet now is for social purposes: finding comments and forums or content from other users, and you don’t really need to be untracked to do all that.
A megacorp’s main motivation for tracking your identity is to sell you shit or sell your data to other people who want to sell you things. But if you’re using AI the amount of ads and SEO spam that you have to sift through will dramatically reduce, rendering most of those efforts pointless.
And most people aren’t using the internet like in the old days: stumbling across quaint cozy boutique websites made by hobbyists about some favorite topic. People just jump on social platforms and consume content until satisfied.
There is no money to be made anymore in mass web scraping at scale with impersonated clients, it’s all been consumed.
Re: Curl-Impersonate
#45The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyo…
> I can't help but feel like these are the dying breaths of the open Internet though I agree with the over zealous tracking by the megacorps but this is also due to bad actors, I work for a financial company and the amount of API abuse, ATO, DDoS, nefarious bot traffic, etc. we see on a daily basis is absolutely insane
Much of this "bad actor" activity is actually customer needs left hanging - for either the customer to automate herself or other companies to fill the gap to create value that's not envisioned by the original company.
I'm guessing investors actually like a healthy dose of open access and a healthy dose of defence. We see them (YC, as an example) betting on multiple teams addressing the same problem. The difference is their execution, the angle they attack.
If, say, the financial company you work for is capable in both product and technical aspect, I assume it leaves no gap. It's the main place to access the service and all the side benefits.
Re: Curl-Impersonate
#46In case anyone is interested, I created something similar but for python(using chromium's network stack) https://github.com/lagenar/python-cronet I'm looking for help to create the build for windows.
Any reason you didn’t use https://github.com/lexiforest/curl_cffi ?
I wanted to try a diffent approach which is to use chromium's network stack directly instead of patching curl to impersonate it. In this case you're using the real thing so it's a bit easier to maintain when there are changes in the fingerprint.
Re: Curl-Impersonate
#47Earlier quoted context omitted.
I can tell you about my experience with blocking traffic from scalpers bots that were very active during pandemic. All requests produced by those bots were valid ones, nothing that could be flagged by tools like fail2ban etc (my assumption is that it would be the same for financial systems). Any blocking or rate limiting by IP is useless, we saw about 2-3 requests per minute per IP, and those actors had access to rid…
Blocking scalper bot traffic by any means, be it by source or certified identification seems a lost cause, i.e. not possible because it can always be circumvented. Why did you not have that filter at point of sale instead? I'm sure there are reasons, but to have a battery of captchas and a limit on purchases per credit card seems on the surface much more sturdy. And it doesn't require that everyone browsing the inter…
The product they tried to buy what not in stock anyways, but their strategy was to constantly try anyways, so in case it would become in stock they would be the first to get it.
It was all for guest checkout, so no address yet to validate nor credit card.
Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements.
As stated before the main reason we needed to block it was volume of the traffic, you migh imagine identical scenario for dealing with DDoS attack.
Re: Curl-Impersonate
#48Earlier quoted context omitted.
Blocking scalper bot traffic by any means, be it by source or certified identification seems a lost cause, i.e. not possible because it can always be circumvented. Why did you not have that filter at point of sale instead? I'm sure there are reasons, but to have a battery of captchas and a limit on purchases per credit card seems on the surface much more sturdy. And it doesn't require that everyone browsing the inter…
The product they tried to buy what not in stock anyways, but their strategy was to constantly try anyways, so in case it would become in stock they would be the first to get it. It was all for guest checkout, so no address yet to validate nor credit card. Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements. As stated before the main reason…
> Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements.
That doesn't compute... Captcha is almost always used in such setups.
It also looks like you could just offer an API endpoint which would return if the article is in stock or not, or even provide a webhook. Why fight them? Just make the resource usage lighter.
I'm curious now though what the articles were, if you are at liberty to share?
Re: Curl-Impersonate
#49What are some example sites where this is both necessary and sufficient? In my experience sites with serious anti-bot protection basically always have JavaScript-based browser detection, and some are capable of defeating puppeteer-extra-plugin-stealth even in headful mode. I doubt sites without serious anti-bot detection will do TLS fingerprinting. I guess it is useful for the narrower use case of getting a short-liv…
There are sites that will block curl and python-requests completely, but will allow curl-impersonate. IIRC, Amazon is an example that has some bot protection but it isn't "serious".
In most cases this is just based on user agent. It's widespread enough that I just habitually tell requests not to set a User Agent at all (these aren't blocked, but if the UA contains "python" it is).
Re: Curl-Impersonate
#50Earlier quoted context omitted.
But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to appl…
I can tell you about my experience with blocking traffic from scalpers bots that were very active during pandemic. All requests produced by those bots were valid ones, nothing that could be flagged by tools like fail2ban etc (my assumption is that it would be the same for financial systems). Any blocking or rate limiting by IP is useless, we saw about 2-3 requests per minute per IP, and those actors had access to rid…
why not charge people? This is the only solution I can think of.