Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping: Bypassing “403 Forbidden,” captchas, and more

sangaline.com

Post reply on HN
Print view View original

171–180 of 232 posts

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#171

Post by nuggien »

dsacco wrote: post #68
jedberg wrote: post #63

Earlier quoted context omitted.

How do you deal with the issue that most mobile apps have a baked in security key for their private API? Or am I being naive to think that most apps have that?

You reverse engineer the application, or you run it in a debugger. If the app features certificate pinning to block MITM eavesdropping through your own proxy, you either use one of the XPosed Framework libraries that removes it on the fly in a process hook, or you decompile the app, return-void the GetTrustedClient, GetTrustedServer, AcceptedIssuers, etc. functions. If it features HMAC signing, you decompile the app,…

If they're smart enough to collect data in the app to detect when a request to their api is actually coming from their app (versus one coming from curl or from a script), then your reverse engineering problem just went from painful to unprofitable (i.e. not worth your time).

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#172

Post by Rebelgecko »

lhnz wrote: post #170
phs318u wrote: post #159

Earlier quoted context omitted.

Perfectly feasible technically. However aren't you stepping into a legal minefield? After all, it's a "private" API. Furthermore, since you're explicitly going around the hurdles they've laid out for the public (captchas, T&C), you're surely painting a target on your head? Note, I'm not debating whether this is a good or bad thing, just that in the current environment this is surely a legally dodgy manoeuvre.

Has anybody ever been sued for scraping a public website?

Weev spent about a year in jail for doing it

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#173

Post by callmeal »

brilliantcode wrote: post #146
CWuestefeld wrote: post #144

Earlier quoted context omitted.

Having been the victim of a VERY badly behaved scraper, I'm willing to listen to this. When that "attack" was going on, we talked about that very thing, if the scraper would only identify himself. (we were able to identify the actual culprit, and circumstantial evidence suggested they were going after our complete price list for a client) The cost of the bad scraper was pretty significant. They were hitting us as har…

Story doesn't add up. First, it's very hard to pull off a DDOS attack using Tor. The most you could get would be less than someone repeatedly pressing refresh every second. This is because if you hit the same domain repeatedly the network will flag and throttle you. How bad was your server configuration that it would choke if somebody tried to scrape it? Was this running on a dreamhost $10/year server or something? T…

CWuestefeld wrote "They were hitting us as hard as they could, through TOR nodes and various cloud providers."

I think you missed the second part of that sentence. I must admit that I worked for a company that did that to scrape a well known business networking site....

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#174

Post by corford »

foxylion wrote: post #51

I'm curious what others use to scrape modern (javascript based) web applications. The old web (html and links) work fine with tools like Scrapy, but for modern applications which rely on javascript this does no longer work. For my last project I used a chrome plugin which controlled the browsers url locations and clicks. Results where transmitted to a backend server. New jobs (clicks, change urls) where retrieved fro…

Casperjs with Slimerjs and/or Phantomjs work well

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#175

Post by codedokode »

hsod wrote: post #88
codedokode wrote: post #38

Earlier quoted context omitted.

I think such laws are wrong. Scraping is what Google and Web Archive does and it serves good purposes. For example, one can make an application that compares prices for the same item at different internet shops and helps to find the cheapest offer. I don't understand what's wrong with downloading the information that is published on a public web server. That is what that server was made for in the first place. Of cou…

I disagree. The person who owns the server should get to decide who has access and under what circumstances. Joe Scraper, having invested nothing, has no claim or rights to it whatsoever. Furthermore, the fact that the server gives 200 responses is not sufficient implied permission IF a no-scraping policy has been communicated in some other way such as robots.txt or (clearly communicated) TOS. The techno-nihilist arg…

The person who owns a server can restrict access by adding a login form and creating user accounts. Or taking other measures (for example, banning IP networks or countries or implementing a captcha).

I think that the only thing that has to be regulated is the load (number of request per unit of time) on the server. So that it doesn't prevent server from serving pages to other visitors.

Regarding robots.txt, I am not sure if ignoring it should have legal consequences. It is just a hint to robots, don't visit these pages or we will ban you.

If there is no harm done to the server I don't see any problem with scraping.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#176

Post by corford »

fiatjaf wrote: post #160

What if the target page is blocking by IP address and if even with 20 different IP addresses you wouldn't be able to fetch all the data you need in a month?

Professional proxy services. Price, IP pool size and quality vary hugely but if you're not trying to scrape an aggressively defended target and don't need to make more than a handful of requests per second, 100K IPs will usually be more than enough to circumvent most rate limits and a pool that size can be rented for under $100/month.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#177

Post by codedokode »

garethadams wrote: post #89
codedokode wrote: post #38

Earlier quoted context omitted.

I think such laws are wrong. Scraping is what Google and Web Archive does and it serves good purposes. For example, one can make an application that compares prices for the same item at different internet shops and helps to find the cheapest offer. I don't understand what's wrong with downloading the information that is published on a public web server. That is what that server was made for in the first place. Of cou…

That's fine as a discussion starter, but thinking a law is wrong isn't a reason to break it – if that's what you're suggesting. That argument often turns into "But I probably won't get caught" which is a similarly weak defence.

I am not suggesting to break a law (I don't live in US so that law doesn't apply to me anyway). I just try to point out that it is wrong and serves only publishers' interests.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#178

Post by corford »

skinnymuch wrote: post #166
jordif wrote: post #151

Good article! I been doing scraping for the last 10 years and I've seen a lots of differents things to try to avoid us. Also, I'm in the other side protecting websites to ban scrapers, so funny!

I'm in the same position for the first time (protecting against scraping) and honestly I'm kind of blind right now. Which is weird because of how much scraping I've done (okay not that much). Any tips or tricks or blogs you know of off the top of your head for protecting your site?

Virtually everything can be easily defeated. The only outfit I've consistently seen put up a good fight is Distil. They do it by acting a little like Cloudflare. They put their servers in front of your www facing endpoints and use ML to mine their global client traffic to identify bot signals (aided by some aggressive in-browser javascript fingerprinting).

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#179

Post by codedokode »

hsod wrote: post #142
brilliantcode wrote: post #139

Earlier quoted context omitted.

This feels a lot like the "downloading mp3 is stealing". If you don't want people stealing your music, don't put it other people's hands. The minute you release it to the world, it can't be reversed. See streisand effect. Likewise, you cannot place burden on your visitors to read and analyze ToS with their lawyers and submit an official request via fax. If you don't want people access your server, put it behind a pay…

Seems like your argument is based on the idea that it's unreasonable to expect the end user to understand the terms under which a service is offered. And that's fair in some cases, but it's also kind of a cop out. "How was I supposed to know they didn't want me to scrape it? I'm just an innocent passerby dropping bits on a wire" is bullshit in many, many cases. You do know, or at least could easily find out if you wa…

Scraping a website can be compared to only downloading mp3 file, not redistributing it to other people.

And I don't see how electronic ToS have any legal power. It is not a contract one has signed before visiting a site. I think ToS should only bind the website owner, not its visitors.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#180

Post by JustSomeNobody »

charlesdm wrote: post #127
diminoten wrote: post #115

Earlier quoted context omitted.

Once you do the one-time work of pulling out the key, you can just add something like, "secret_key=foobar" to your requests, and you're back to happily parsing JSON. If they keep changing it up, I'm sure you could automate the decompiling process. The reality is that this technique is security by obscurity at its core, and is therefore never going to succeed.

Skype is probably one example where it took developers 10+ years to figure out how the app worked.

Did it take that long to do it or did it take that long for someone to care to do it? I mean, it's Skype.
Post reply on HN