> The resulting curl looks, from a network perspective, identical to a real browser. How close is it? If I ran wireshark, would the bytes be exactly the same in the exact same packets?
You can check your fingerprint on https://tls.peet.ws
Curl-Impersonate
31–40 of 114 posts
Re: Curl-Impersonate
#32I think we should list the sites where this fingerprinting is done. I have a suspicion that Microsoft does it for conditional access policies but I am not sure of other services.
Re: Curl-Impersonate
#33Earlier quoted context omitted.
> I can't help but feel like these are the dying breaths of the open Internet though I agree with the over zealous tracking by the megacorps but this is also due to bad actors, I work for a financial company and the amount of API abuse, ATO, DDoS, nefarious bot traffic, etc. we see on a daily basis is absolutely insane
But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to appl…
A big problem is that where we have a good solution you'll lose if you insist on that solution but other people get away with doing something that's crap but customers like better. We often have to mandate a poor solution that will be tolerated because if we mandate the better solution it will be rejected, and if we don't mandate anything the outcomes are far worse.
Today for example I changed energy company†. I made a telephone call, from a number the company has never seen before. I told them my name (truthfully but I could have lied) and address (likewise). I agreed to about five minutes of parameters, conditions, etc. and I made one actual meaningful choice (a specific tariff, they offer two). I then provided 12 digits identifying a bank account (they will eventually check this account exists and ask it to pay them money, which by default will just work) and I'm done.
Notice that anybody could call from a burner and that would work too. They could move Aunt Sarah's energy to some random outfit, assign payments to Jim's bank account, and cause maybe an hour of stress and confusion for both Sarah and Jim when months or years later they realise the problem.
We know how to do this properly, but it would be high friction and that's not in the interests of either the "energy companies" or the politicians who created this needlessly complicated "Free Market" for energy. We could abolish that Free Market, but again that's not in their interests. So, we're stuck with this waste of our time and money, indefinitely.
There have been simpler versions of this system, which had even worse outcomes. They're clumsier to use, they cause more people to get scammed AND they result in higher cost to consumers, so that's not great. And there are better systems we can't deploy because in practice too few consumers will use them, so you'd have 0% failure but lower total engagement and that's what matters.
† They don't actually supply either gas or electricity, that's a last mile problem solved by a regulated monopoly, nor do they make electricity or drill for gas - but they do bill me for the gas and electricity I use - they're an artefact of Capitalism.
Re: Curl-Impersonate
#34What are some example sites where this is both necessary and sufficient? In my experience sites with serious anti-bot protection basically always have JavaScript-based browser detection, and some are capable of defeating puppeteer-extra-plugin-stealth even in headful mode. I doubt sites without serious anti-bot detection will do TLS fingerprinting. I guess it is useful for the narrower use case of getting a short-liv…
CloudFlare offers it. Even if it's not used for blocking it might be used for analytics or threat calculations, so you might get hit later.
Re: Curl-Impersonate
#35I think we should list the sites where this fingerprinting is done. I have a suspicion that Microsoft does it for conditional access policies but I am not sure of other services.
We cannot really list them, as 90% of the time, it's not the websites themselves, it's their WAF. And there is a trend toward most company websites to be behind a WAF nowadays to avoid 1) annoying regulations (US companies putting geoloc on their websites to avoid EU cookie regulations) and 2) DDoS.
It's now pretty common to have cloudflare, AWS, etc WAFs as main endpoints, and these do anti bots (TLS fingerprinting, header fingerprinting, Javascript checks, capt has, etc).
Re: Curl-Impersonate
#36Earlier quoted context omitted.
> I can't help but feel like these are the dying breaths of the open Internet though I agree with the over zealous tracking by the megacorps but this is also due to bad actors, I work for a financial company and the amount of API abuse, ATO, DDoS, nefarious bot traffic, etc. we see on a daily basis is absolutely insane
But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to appl…
I can tell you about my experience with blocking traffic from scalpers bots that were very active during pandemic.
All requests produced by those bots were valid ones, nothing that could be flagged by tools like fail2ban etc (my assumption is that it would be the same for financial systems).
Any blocking or rate limiting by IP is useless, we saw about 2-3 requests per minute per IP, and those actors had access to ridiculous number of large CIDRs, blocking any IP caused it instantly replace it with another.
blocking by AS number was also mixed bag, as this list growed up really quickly, most of that were registered to suspicious looking Gmail addresses. (I feel that such activity might own significant percentage of total ipv4 space)
This was basically cat and mouse game of finding some specific characteristic in requests that matches all that traffic and filtering it, but the other side would adapt next day or on Sunday.
aggregated amount of traffic was in range of 2-20k r/s to basically heaviest endpoint in the shop, with was the main reason we needed to block that traffic (it generated 20-40x load of organic traffic)
cloudflare was also not really successful with default configuration, we had to basically challenge everyone by default with whitelist of most common regions from where we expected customers.
So best solution is to track everyone and calculate long term reputation.
Re: Curl-Impersonate
#37In case anyone is interested, I created something similar but for python(using chromium's network stack) https://github.com/lagenar/python-cronet I'm looking for help to create the build for windows.
Any reason you didn’t use https://github.com/lexiforest/curl_cffi?
Re: Curl-Impersonate
#38Earlier quoted context omitted.
But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to appl…
> E.g. a large fai2ban rule to just ban anything that attempts to HTTP GET /admin.php or /phpmyadmin etc, even just once, gets rid of almost all nefarious bot traffic. unfortunately fail2ban wouldn't even make a dent in the attack traffic hitting the endpoints in my day-to-day work, these are attackers utilizing residential proxy infrastructure that are increasingly capable of solving JS/client-puzzle challenges.. th…
we see the same thing, also with a financial company, the most successful strategies we’ve seen is making stuff like this extremely expensive for whoever it is if we see it, and they stop or slow down to a point it becomes not worth it and they move on. sometimes that’s really all you can do without harming legit traffic.
Re: Curl-Impersonate
#39What is the use case? If you have to read data from one specific website which uses handshake info to avoid being read by software? When I have to do HTTP requests these days, I default to a headless browser right away, because that seems to be the best bet. Even then, some website are not readable because they use captchas and whatnot.
> I default to a headless browser Headless browsers consume orders of magnitude more resources, and execute far more requests (e.g. fetching images) than a common webscraping job would require. Having run webscraping at scale myself, the cost of operating headless browsers made us only use them as a last resort.
So you maintain a table of domains and how to access them?
How do you build that table and keep it up to date? Manually?
Re: Curl-Impersonate
#40Earlier quoted context omitted.
> I can't help but feel like these are the dying breaths of the open Internet though I agree with the over zealous tracking by the megacorps but this is also due to bad actors, I work for a financial company and the amount of API abuse, ATO, DDoS, nefarious bot traffic, etc. we see on a daily basis is absolutely insane
But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to appl…
That's not the same type of bot net. Fail 2 ban simply is not going to work when you have a popular unauthenticated endpoint. You have hundreds of thousands of rps spread across thousands of legitimate networks that. The requests are always modified to look legitimate in a never ending game of whack-a-mole.
You wind up having to use things like tls fingerprinting with other heuristics to identify what to traffic to reject. These all take engineering hours and require infrastructure. It is SO MUCH SIMPLER to require auth and reject everything else outright.
I know that the BigCo's want to track us and you originally mentioned tracking not auth. But my point is yeah, they have malicious reasons for locking things down, but there are legitimate reasons too.