Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping: Bypassing “403 Forbidden,” captchas, and more

sangaline.com

Post reply on HN
Print view View original

221–230 of 232 posts

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#221

Post by jakubbalada »

brilliantcode wrote: post #215
jakubbalada wrote: post #186

Earlier quoted context omitted.

Both - developers on a free plan using own RSS for sites without one and business people (mainly startups) building their products on top of Apifier. Typical use is an aggregator that needs common API for all partners who are not able to provide it. So they have running API on Apifier in an hour. It might break once in a while - than you have to update your crawler (not that often if you use internal AJAX calls).

I see, so there's not much value beyond startups and bootstrappers. I feel like it's a hard sell to enterprises. Scraping is viewed inferior to an API so it makes sense for enterprises to just pay the target website for access to the data.

It's also hard to get direct access to the data.

But you're right it's a hard sell to enterprises although we have some (e.g. real estate developer creating pricing maps)

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#222

Post by grkvlt »

nuggien wrote: post #171
dsacco wrote: post #68

Earlier quoted context omitted.

You reverse engineer the application, or you run it in a debugger. If the app features certificate pinning to block MITM eavesdropping through your own proxy, you either use one of the XPosed Framework libraries that removes it on the fly in a process hook, or you decompile the app, return-void the GetTrustedClient, GetTrustedServer, AcceptedIssuers, etc. functions. If it features HMAC signing, you decompile the app,…

If they're smart enough to collect data in the app to detect when a request to their api is actually coming from their app (versus one coming from curl or from a script), then your reverse engineering problem just went from painful to unprofitable (i.e. not worth your time).

I'm not sure I understand this. The point of the reverse engineering is to simulate (exactly) requests from the app, using your script, thus making it impossible for them to detect. And the data collected in the app has no bearing on your scripted calls to the API outside the app.

I suppose I can envisage a scenario where the app sends a counter as a request id, and my script will then send the next value of the counter as its id, causing the next request from the app to re-use a counter value and thus fail, but the server API and the app have no way of knowing this is due to my script and not, say, network issues, and therefore it should still not affect my reverse engineering abilities...

Maybe, taking this further, the app could have baked in a certain number of unique, opaque 'request tokens' that are valid for one API call only, and when my script has used all of them it will cease to work, and in doing so cause my copy of the app to become useless, but again, not an insurmountable barrier.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#223

Post by kbenson »

corford wrote: post #178
skinnymuch wrote: post #166

Earlier quoted context omitted.

I'm in the same position for the first time (protecting against scraping) and honestly I'm kind of blind right now. Which is weird because of how much scraping I've done (okay not that much). Any tips or tricks or blogs you know of off the top of your head for protecting your site?

Virtually everything can be easily defeated. The only outfit I've consistently seen put up a good fight is Distil. They do it by acting a little like Cloudflare. They put their servers in front of your www facing endpoints and use ML to mine their global client traffic to identify bot signals (aided by some aggressive in-browser javascript fingerprinting).

Yeah, Distil is the first outfit I've encountered where they've got the model to make it really hard to reliably bypass. It comes down to "I can spend a significant amount of time trying to bypass this, and I would, but they would likely identify and block me again within a few weeks at most.", and it's not worth it when it's only part of what I need to do to scrap some data, and it's their entire job, and they can afford to hire multiple people.

The economics are in their favor, and I make it a point not to fight economics when I recognize them, it's rarely sustainable.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#225

Post by elchief »

skinnymuch wrote: post #167
elchief wrote: post #86

Earlier quoted context omitted.

We use HTMLUnit. Works pretty well. Not super fast, but you want to scrape individual sites at a moderate rate anyway

Have you run into issues? I'd think HTMLUnit isn't robust enough and it's "browser" Ian limiting?

It's got a couple of idiosyncrasies but works well in general. Barfs out too much log info in general. XPath 1 is limiting, but can use Saxon if need to

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#226

Post by nuggien »

grkvlt wrote: post #222
nuggien wrote: post #171

Earlier quoted context omitted.

If they're smart enough to collect data in the app to detect when a request to their api is actually coming from their app (versus one coming from curl or from a script), then your reverse engineering problem just went from painful to unprofitable (i.e. not worth your time).

I'm not sure I understand this. The point of the reverse engineering is to simulate (exactly) requests from the app, using your script, thus making it impossible for them to detect. And the data collected in the app has no bearing on your scripted calls to the API outside the app. I suppose I can envisage a scenario where the app sends a counter as a request id, and my script will then send the next value of the coun…

Never assume something is impossible. Tokens/counters that are generated at runtime inside the app is a start at countering bots. But there are MUCH more advanced techniques and big businesses that are built upon helping mobile and webapps detect bots and other scripted requests.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#227

Post by ic3cold »

Have you seen the sentry antirobot system I can't remember the name exactly but it's a hosted solution that randomly displays captchas, when it senses suspicious(robots) crawling. It's a nightmare, because after you solve 1 captcha it can display 4 more one after the other. They also ban your IP, so oyu need IP rotators. Any workarounds? ic3cold

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#228

Post by skinnymuch »

elchief wrote: post #225
skinnymuch wrote: post #167

Earlier quoted context omitted.

Have you run into issues? I'd think HTMLUnit isn't robust enough and it's "browser" Ian limiting?

It's got a couple of idiosyncrasies but works well in general. Barfs out too much log info in general. XPath 1 is limiting, but can use Saxon if need to

Holy crap! Xpath 1 is still being used for it? I actually have no clue what the differences are between xpath versions but I just assumed everyone is on xpath 2.

I guess my other question is - have you run into any situations where the JavaScript parsing or browser rendering wasn't good enough?

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#229

Post by skinnymuch »

corford wrote: post #178
skinnymuch wrote: post #166

Earlier quoted context omitted.

I'm in the same position for the first time (protecting against scraping) and honestly I'm kind of blind right now. Which is weird because of how much scraping I've done (okay not that much). Any tips or tricks or blogs you know of off the top of your head for protecting your site?

Virtually everything can be easily defeated. The only outfit I've consistently seen put up a good fight is Distil. They do it by acting a little like Cloudflare. They put their servers in front of your www facing endpoints and use ML to mine their global client traffic to identify bot signals (aided by some aggressive in-browser javascript fingerprinting).

Interesting thanks.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#230

Post by skinnymuch »

jordif wrote: post #218
skinnymuch wrote: post #166

Earlier quoted context omitted.

I'm in the same position for the first time (protecting against scraping) and honestly I'm kind of blind right now. Which is weird because of how much scraping I've done (okay not that much). Any tips or tricks or blogs you know of off the top of your head for protecting your site?

After the years, I've arrived at the conclusion that everything can be scrapped. What you have to do is try to put as many walls as you can. But if someone really wants to crawl your site, with the right knowledgement he will able to do it despite of all your walls.

Yes that's what I've assumed as well
Post reply on HN