Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Curl-Impersonate

github.com

Post reply on HN
Print view View original

71–80 of 114 posts

Re: Curl-Impersonate

#71

Post by Habgdnv »

octocop wrote: post #25
cle wrote: post #16

The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyo…

"I have nothing to hide" will eventually spread to everyone. Very unfortunate.

The answer is simple: I have something to hide. I have many things to hide actually. Nothing of these things is illegal currently but I still have many things to hide. And if I have something to hide - I can be worried about many things.

Re: Curl-Impersonate

#72

Post by sangnoir »

at0mic22 wrote: post #62
adastral wrote: post #20

Earlier quoted context omitted.

> I default to a headless browser Headless browsers consume orders of magnitude more resources, and execute far more requests (e.g. fetching images) than a common webscraping job would require. Having run webscraping at scale myself, the cost of operating headless browsers made us only use them as a last resort.

Blocking all image/video/CSS requests is the rule of thumb when working with headless browsers via CDP

Speaking as a person who has played on both offense and defense: this is a heuristic that's not used frequently enough by defenders. Clients that load a single HTML/JSON endpoint without loading css or image resources associated with the endpoints are likely bots (or user agents with a fully loaded cache, but defenders control what gets cached by legit clients and how). Bot data thriftiness is a huge signal.

Re: Curl-Impersonate

#74

Post by at0mic22 »

sangnoir wrote: post #72
at0mic22 wrote: post #62

Earlier quoted context omitted.

Blocking all image/video/CSS requests is the rule of thumb when working with headless browsers via CDP

Speaking as a person who has played on both offense and defense: this is a heuristic that's not used frequently enough by defenders. Clients that load a single HTML/JSON endpoint without loading css or image resources associated with the endpoints are likely bots (or user agents with a fully loaded cache, but defenders control what gets cached by legit clients and how). Bot data thriftiness is a huge signal.

As a high load system engineer you'd want to offload asset serving to CDN which makes detection slightly more complicated. The easy way is to attach an image onload handler with client js, but that would give a high yield of false positives. I personally have never seen such approach and doubt its useful for many concerns.

Re: Curl-Impersonate

#75

Post by 38 »

jandrese wrote: post #56

The build scripts on this repo seem a bit cursed. It uses autotools but has you build them in a subdirectory. The default built target is a help text instead of just building the project. When you do use the listed build target it doesn't have the dependencies set up correctly so you have to run it like 6 times to get to the point where it is building the application. Ultimately I was not able to get it to build beca…

that's exactly why I stopped using C/C++. building is many times a nightmare, and the language teams seems to have no interest in improving the situation

Re: Curl-Impersonate

#76

Post by geysersam »

miki123211 wrote: post #65
bornfreddy wrote: post #48

Earlier quoted context omitted.

> Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements. That doesn't compute... Captcha is almost always used in such setups. It also looks like you could just offer an API endpoint which would return if the article is in stock or not, or even provide a webhook. Why fight them? Just make the resource usage lighter. I'm curious now though wha…

> Why fight them? Just make the resource usage lighter. Because you presumably want real, returning customers, and that means those customers need to get a chance at buying those products, instead of them being scooped up by a scalper the millisecond they appear on the website.

Sounds like a dream having customers scooping up your products the millisecond the appear on the website. They should increase their prices.

Re: Curl-Impersonate

#77

Post by codingminds »

Szpadel wrote: post #36
berkes wrote: post #26

Earlier quoted context omitted.

But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to appl…

I can tell you about my experience with blocking traffic from scalpers bots that were very active during pandemic. All requests produced by those bots were valid ones, nothing that could be flagged by tools like fail2ban etc (my assumption is that it would be the same for financial systems). Any blocking or rate limiting by IP is useless, we saw about 2-3 requests per minute per IP, and those actors had access to rid…

I've learned that Akamai has a service that deals with this specific problem, maybe this might interest you as well: https://www.akamai.com/products/content-protector

Re: Curl-Impersonate

#78

Post by zzo38computer »

sangnoir wrote: post #72
at0mic22 wrote: post #62

Earlier quoted context omitted.

Blocking all image/video/CSS requests is the rule of thumb when working with headless browsers via CDP

Speaking as a person who has played on both offense and defense: this is a heuristic that's not used frequently enough by defenders. Clients that load a single HTML/JSON endpoint without loading css or image resources associated with the endpoints are likely bots (or user agents with a fully loaded cache, but defenders control what gets cached by legit clients and how). Bot data thriftiness is a huge signal.

Even legitimate users might want to disable CSS and pictures and whatever, and I often do when I just want to read the document.

Blind users also might have no use for the pictures, and another possibility is if the document is longer than the screen so the picture is out of view then the user might program the client software to use lazy loading, etc.

Re: Curl-Impersonate

#79

Post by sangnoir »

at0mic22 wrote: post #74
sangnoir wrote: post #72

Earlier quoted context omitted.

Speaking as a person who has played on both offense and defense: this is a heuristic that's not used frequently enough by defenders. Clients that load a single HTML/JSON endpoint without loading css or image resources associated with the endpoints are likely bots (or user agents with a fully loaded cache, but defenders control what gets cached by legit clients and how). Bot data thriftiness is a huge signal.

As a high load system engineer you'd want to offload asset serving to CDN which makes detection slightly more complicated. The easy way is to attach an image onload handler with client js, but that would give a high yield of false positives. I personally have never seen such approach and doubt its useful for many concerns.

Unless organization policy forces you to, you do not have to put all resources behind a CDN. As a matter of fact, getting this heuristic to work requires a non-optimal caching strategy of one or more real or decoy resources - CDN or not. "Easy" is not an option for the bot/anti-bot arms race, all the low hanging fruit is now gone when fighting a determined adversary on either end.

> I personally have never seen such approach and doubt its useful for many concerns.

It's an arms race and defenders are not keen on sharing their secret sauce, though I can't be the only one who thought of this rather basic bot characteristic, multiple abuse trams probably realized this decades ago. It works pretty well against the low-resource scrapers with fakes UA strings and all the right TLS handshakes. It won't work against the headless browsers that costs scrapers more in resources and bandwidth, and there are specific countermeasures for headless browsers [1], and counter-countermeasures. It's a cat and mouse game.

1. e.g. Mouse movement, as made famous as ine signal evaluated by Google's reCAPTCHA v2, monitor resolution & window size and position, and Canvas rendering, all if which have been gradually degraded by browser anti-fingerprinting efforts. The bot war is fought on the long tail.

Re: Curl-Impersonate

#80

Post by shaky-carrousel »

Szpadel wrote: post #54
bornfreddy wrote: post #48

Earlier quoted context omitted.

> Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements. That doesn't compute... Captcha is almost always used in such setups. It also looks like you could just offer an API endpoint which would return if the article is in stock or not, or even provide a webhook. Why fight them? Just make the resource usage lighter. I'm curious now though wha…

We had captcha, but it was at later stage of the checkout process. This API endpoint needed to work from cached pages, so it could not contain any dynamic state in request. Some bots checked product page where we had info if product is in stock (although they tried heavenly to bypass any caches by putting garbage in URL). This kind of bots also scaled instantly to thousands checkout requests when product become avail…

Hm, is probably too late, but you could have implemented in your API calls some kind of proof of work. Something that's not too onerous for a casual user but it is hard for someone trying multiple requests.
Post reply on HN