Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Curl-Impersonate

github.com

Post reply on HN
Print view View original

61–70 of 114 posts

Re: Curl-Impersonate

#61

Post by remram »

oefrha wrote: post #18

What are some example sites where this is both necessary and sufficient? In my experience sites with serious anti-bot protection basically always have JavaScript-based browser detection, and some are capable of defeating puppeteer-extra-plugin-stealth even in headful mode. I doubt sites without serious anti-bot detection will do TLS fingerprinting. I guess it is useful for the narrower use case of getting a short-liv…

Those JavaScript scripts often get data from some API, and it's that API that will usually be behind some fingerprinting wall.

Re: Curl-Impersonate

#62

Post by at0mic22 »

adastral wrote: post #20
TekMol wrote: post #11

What is the use case? If you have to read data from one specific website which uses handshake info to avoid being read by software? When I have to do HTTP requests these days, I default to a headless browser right away, because that seems to be the best bet. Even then, some website are not readable because they use captchas and whatnot.

> I default to a headless browser Headless browsers consume orders of magnitude more resources, and execute far more requests (e.g. fetching images) than a common webscraping job would require. Having run webscraping at scale myself, the cost of operating headless browsers made us only use them as a last resort.

Blocking all image/video/CSS requests is the rule of thumb when working with headless browsers via CDP

Re: Curl-Impersonate

#63

Post by miki123211 »

berkes wrote: post #26
throwaway99210 wrote: post #19

Earlier quoted context omitted.

> I can't help but feel like these are the dying breaths of the open Internet though I agree with the over zealous tracking by the megacorps but this is also due to bad actors, I work for a financial company and the amount of API abuse, ATO, DDoS, nefarious bot traffic, etc. we see on a daily basis is absolutely insane

But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to appl…

This depends on what you're fighting.

If you're fighting adversaries that go for scale, AKA trying to hack as many targets as possible, mostly low-sophistication, using techniques requiring 0 human work and seeing what sticks, yes, blocking those simple techniques works.

Those attackers don't ever expect to hack Facebook or your bank, that's just not the business they're in. They're fine with posting unsavory ads on your local church's website, blackmailing a school principal with the explicit pictures he stores on the school server, or encrypting all the data on that server and demanding a ransom.

If your company does something that is specifically valuable to someone, and there are people whose literal job it is to attack your company's specific systems, no, those simple techniques won't be enough.

If you're protecting a Church with 150 members, the simple techniques are probably fine, if you're working for a major bank or a retailer that sells gaming consoles or concert tickets, they're laughably inadequate.

Re: Curl-Impersonate

#64

Post by at0mic22 »

jandrese wrote: post #56

The build scripts on this repo seem a bit cursed. It uses autotools but has you build them in a subdirectory. The default built target is a help text instead of just building the project. When you do use the listed build target it doesn't have the dependencies set up correctly so you have to run it like 6 times to get to the point where it is building the application. Ultimately I was not able to get it to build beca…

Played this game and switched to prebuilt libraries. Think builder docker images have also been broken for a while.

Re: Curl-Impersonate

#65

Post by miki123211 »

bornfreddy wrote: post #48
Szpadel wrote: post #47

Earlier quoted context omitted.

The product they tried to buy what not in stock anyways, but their strategy was to constantly try anyways, so in case it would become in stock they would be the first to get it. It was all for guest checkout, so no address yet to validate nor credit card. Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements. As stated before the main reason…

> Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements. That doesn't compute... Captcha is almost always used in such setups. It also looks like you could just offer an API endpoint which would return if the article is in stock or not, or even provide a webhook. Why fight them? Just make the resource usage lighter. I'm curious now though wha…

> Why fight them? Just make the resource usage lighter.

Because you presumably want real, returning customers, and that means those customers need to get a chance at buying those products, instead of them being scooped up by a scalper the millisecond they appear on the website.

Re: Curl-Impersonate

#67

Post by miki123211 »

code51 wrote: post #45
throwaway99210 wrote: post #19

Earlier quoted context omitted.

> I can't help but feel like these are the dying breaths of the open Internet though I agree with the over zealous tracking by the megacorps but this is also due to bad actors, I work for a financial company and the amount of API abuse, ATO, DDoS, nefarious bot traffic, etc. we see on a daily basis is absolutely insane

Much of this "bad actor" activity is actually customer needs left hanging - for either the customer to automate herself or other companies to fill the gap to create value that's not envisioned by the original company. I'm guessing investors actually like a healthy dose of open access and a healthy dose of defence. We see them (YC, as an example) betting on multiple teams addressing the same problem. The difference is…

> Much of this "bad actor" activity is actually customer needs left hanging - for either the customer to automate herself or other companies to fill the gap to create value

Sometimes the customer you have isn't the customer you want.

As a bank, you don't want the customers that will try to log in to 1000 accounts, and then immediately transfer any money they find to the Seychelles. As a ticketing platform, you don't want the customers that buy tickets and then immediately sell them on for 4x the price. As a messaging app, you don't want the customers who have 2000 bot accounts and use AI to send hundreds of thousands of spam messages a day. As a social network, you don't want the customers who want to use your platform to spread pro-russian misinformation.

In a sense, those are "customer needs left changing", but neither you nor otherr customers want those needs to be automatible.

Re: Curl-Impersonate

#68

Post by lelandfe »

schnable wrote: post #22
cle wrote: post #16

The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyo…

A lot of the motivation comes from government regulations too. Right now this is mostly in banking, but social media and porn regs are coming too.

PornHub and all of its affiliate sites now block all residents of Alabama, Arkansas, Idaho, Indiana, Kansas, Kentucky, Mississippi, Montana, Nebraska, North Carolina, Texas, Utah, and Virginia (and Florida on Jan 1): https://www.pcmag.com/news/pornhub-blocked-florida-alabama-t...

Child safety, as always, was the sugar that made the medicine go down in freedom-loving USA. I imagine these states' approaches will try to move to the federal level after Section 230 dies an ignominious death.

Keep an eye out for Free Speech Coalition v. Paxton to hit SCOTUS in January: https://www.oyez.org/cases/2024/23-1122

Re: Curl-Impersonate

#69

Post by josephcsible »

throwaway99210 wrote: post #29
berkes wrote: post #26

Earlier quoted context omitted.

But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to appl…

> E.g. a large fai2ban rule to just ban anything that attempts to HTTP GET /admin.php or /phpmyadmin etc, even just once, gets rid of almost all nefarious bot traffic. unfortunately fail2ban wouldn't even make a dent in the attack traffic hitting the endpoints in my day-to-day work, these are attackers utilizing residential proxy infrastructure that are increasingly capable of solving JS/client-puzzle challenges.. th…

Such a rule is a great way to let malicious users lock out a bunch of your legitimate customers. Imagine if someone makes a forum post and includes this in it:

  [img]https://example.com/phpmyadmin/whatever.png[/img]

Re: Curl-Impersonate

#70

Post by sangnoir »

mattpallissard wrote: post #40
berkes wrote: post #26

Earlier quoted context omitted.

But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to appl…

That's not the same type of bot net. Fail 2 ban simply is not going to work when you have a popular unauthenticated endpoint. You have hundreds of thousands of rps spread across thousands of legitimate networks that. The requests are always modified to look legitimate in a never ending game of whack-a-mole. You wind up having to use things like tls fingerprinting with other heuristics to identify what to traffic to r…

> You wind up having to use things like tls fingerprinting

...and we've circled back to the post's subject - a version of curl that impersonates browsers TLS handshake behavior to bypass such fingerprinting.

Post reply on HN