(very rough) ebuild: https://github.com/jakeogh/jakeogh/blob/master/net-misc/curl...
Curl-Impersonate
101–110 of 114 posts
Re: Curl-Impersonate
#102I like this project! Is there a way to request impersonization of the current version of Chrome (or whatever)?
The latest version is a moving target, currently you get the following chrome versions:
$ curl_chrome
curl_chrome100
curl_chrome101
curl_chrome104
curl_chrome107
curl_chrome110
curl_chrome116
curl_chrome119
curl_chrome120
curl_chrome123
curl_chrome124
curl_chrome131
curl_chrome131_android
curl_chrome99
curl_chrome99_androidRe: Curl-Impersonate
#103The build scripts on this repo seem a bit cursed. It uses autotools but has you build them in a subdirectory. The default built target is a help text instead of just building the project. When you do use the listed build target it doesn't have the dependencies set up correctly so you have to run it like 6 times to get to the point where it is building the application. Ultimately I was not able to get it to build beca…
Hi, maintainer here, the whole project is a hack, actually :P
The original repo was already full of hacks, and on top of that, I added more hacks to keep up with the latest browsers. The main purpose of my fork is to serve as a foundation of the python binding, which I think is easier to use. So I haven't tried to make the whole process more streamlined as long as it works on the CI. You can use the prebuilt binaries on the release page, though. I guess I should find some time to clean up the whole thing.
Re: Curl-Impersonate
#104I can't help but think that projects like these shouldn't be posted here, since the enemy is among us. Prodding the bear even more might lead to an acceleration towards the dystopia that others here have already prophesised. The following browsers can be impersonated. ...unfortunately no Firefox to be seen. I've had to fight this too, since I use a filtering proxy. User-agent discrimination should be illegal. One may…
Maintainer here. Curl drops NSS support since like a year ago, which is the SSL engine firefox uses. Without NSS, two special extensions can not be added. And that's why only webkit-based browsers are left.
You can find support for old firefox versions in the original repo.
Re: Curl-Impersonate
#105Earlier quoted context omitted.
"I have nothing to hide" will eventually spread to everyone. Very unfortunate.
The answer is simple: I have something to hide. I have many things to hide actually. Nothing of these things is illegal currently but I still have many things to hide. And if I have something to hide - I can be worried about many things.
"It's not that I have something to hide, there's simply nothing I want to show you."
Re: Curl-Impersonate
#106The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyo…
> ensure you are only doing things they approve of
Absolutely. They might not care about individuals, though. It's their approach to shape "markets". The Apple, Google, Amazon, and Microsoft tax is not inevitable and that's their problem. They will fight toe and nail to keep you locked in, call it "innovation", and even cooperate with governments (which otherwise are their natural enemy in the fight for digital control). It's the people that a) don't care much and b) don't have any options.
In the end, a large share of our wealth is just pulled from us to these ever more ridiculous rent seeking schemes.
Re: Curl-Impersonate
#107The build scripts on this repo seem a bit cursed. It uses autotools but has you build them in a subdirectory. The default built target is a help text instead of just building the project. When you do use the listed build target it doesn't have the dependencies set up correctly so you have to run it like 6 times to get to the point where it is building the application. Ultimately I was not able to get it to build beca…
I hit that too, there is an open bug: https://github.com/lexiforest/curl-impersonate/issues/81
Worked around it by modifying the patch: https://github.com/jakeogh/jakeogh/blob/master/net-misc/curl...
Considering the complexity, this project, and it's upstream parent and grandparent(curl proper) are downright amazing.
Re: Curl-Impersonate
#108Earlier quoted context omitted.
Such a rule is a great way to let malicious users lock out a bunch of your legitimate customers. Imagine if someone makes a forum post and includes this in it: [img]https://example.com/phpmyadmin/whatever.png[/img]
That would be in the body of the request. OP is talking about URLs in the actual request, which is part of the header. While I don't have experience with a great number of WAFs I'm sure sophisticated ones let you be quite specific on where you are matching text to identify bad requests. As an aside, another "easy win" is assuming any incoming HTTP request for a dotfile is malicious. I see constant unsolicitied attemp…
A lot of modern standards rely on .well-known urls to convey abilities, endpoints, related services and so on.
In my case, I never run anything PHP so I'll just plain block out anything PHP (same for python, lua, activedirectory etc). And, indeed, .htaccess, .env etc. A rather large list of hardcoded stuff that gets an instant-ban. It drops the bot-traffic with 90% or more.
These obviously aren't targeted attacks. Protecting against those is another issue alltogether.
Re: Curl-Impersonate
#109Earlier quoted context omitted.
But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to appl…
I can tell you about my experience with blocking traffic from scalpers bots that were very active during pandemic. All requests produced by those bots were valid ones, nothing that could be flagged by tools like fail2ban etc (my assumption is that it would be the same for financial systems). Any blocking or rate limiting by IP is useless, we saw about 2-3 requests per minute per IP, and those actors had access to rid…
TBC: I wasn't saying that F2B is a silver bullet. Not at all.
But that protection depends on the use case. And that in many of my use-cases, a simple f2b with a large hardcoded list of URL paths I guarantee to never have, will drop bot-traffic with 90% or more. The last 10% then split into "hits because the IP is new" and "other, more sophisticated bots". Bots, in those cases are mostly just stupid worms, just trying out known WP exploits, default passwords on often used tools (nextcloud, phpmyadmin, etc) and so on.
I've done something similar with a large list of known harvest/scraper bots, based on their user-agent (the nice ones), or their movements. Nothing complex, just things like "/hidden-page.html that's linked, but hidden with css/js.
And with spam bots, where certain post-requests can only come from repeatedly submitting the contact form.
This, obviously isn't going to give any protection against targeted attacks. Nor will it protect against more sophisticated bots. But in some -in my case, most- use-cases, it's enough to drop bot-traffic significantly.
Re: Curl-Impersonate
#110The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyo…
The disappearance of the third space is killing us.