Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping: Bypassing “403 Forbidden,” captchas, and more

sangaline.com

Post reply on HN
Print view View original

121–130 of 232 posts

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#121

Post by nostrademons »

makmanalp wrote: post #43
Lxr wrote: post #37

I have done a lot of scraping in Python with requests and lxml and never really understood what scrapy offers beyond that. What are the main features that can't be easily implemented manually?

Pluggable parsers, automatically good error handling and spidering functionality (finding and queueing new links to scrape), great logging, progress stats, exports, pause/resume functionality, and a million other goodies that are seemingly "trivial" but really you don't want to rewrite them every time you write a scraper. edit: Especially if your scraping jobs take a LONG time - days and weeks, this stuff is extra ha…

How much benefit does the spidering/progress/pause/resume functionality give if you're not just spidering every link on the site, but have complex logic to determine exactly which links to crawl and in what order? Does Scrapy provide convenient extension hooks to change the crawl algorithm?

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#122

Post by elmigranto »

hsod wrote: post #88
codedokode wrote: post #38

Earlier quoted context omitted.

I think such laws are wrong. Scraping is what Google and Web Archive does and it serves good purposes. For example, one can make an application that compares prices for the same item at different internet shops and helps to find the cheapest offer. I don't understand what's wrong with downloading the information that is published on a public web server. That is what that server was made for in the first place. Of cou…

I disagree. The person who owns the server should get to decide who has access and under what circumstances. Joe Scraper, having invested nothing, has no claim or rights to it whatsoever. Furthermore, the fact that the server gives 200 responses is not sufficient implied permission IF a no-scraping policy has been communicated in some other way such as robots.txt or (clearly communicated) TOS. The techno-nihilist arg…

Is there a law requiring me to read and accept TOS or robots.txt before opening a website? What is the legal status of those? What if TOS says "you owe us $100 for every byte included in this TOS"? What if I have my own TOS about opening their links where they owe me $100 per byte sent?

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#124

Post by matt_wulfeck »

chatmasta wrote: post #52

Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.

This is absolutely genius. I'd love to see a blog post with more information.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#126

Post by charlesdm »

kilroy123 wrote: post #104
dsacco wrote: post #68

Earlier quoted context omitted.

You reverse engineer the application, or you run it in a debugger. If the app features certificate pinning to block MITM eavesdropping through your own proxy, you either use one of the XPosed Framework libraries that removes it on the fly in a process hook, or you decompile the app, return-void the GetTrustedClient, GetTrustedServer, AcceptedIssuers, etc. functions. If it features HMAC signing, you decompile the app,…

Know of any good guides on doing this? I want to do this for an app right now actually.

Android tends to be easier to decompile if you want to discover stored keys, etc. I've done it a few times by using Charles on desktop + setting up a proxy to run my iOS connection through my laptop, and then running the app on mobile.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#127

Post by charlesdm »

diminoten wrote: post #115
sametmax wrote: post #98

Earlier quoted context omitted.

The initial idea was to make your life simpler by parsing JSON instead of HTML. Now we are decompiling binaries. Somewhere on the way, we got lost.

Once you do the one-time work of pulling out the key, you can just add something like, "secret_key=foobar" to your requests, and you're back to happily parsing JSON. If they keep changing it up, I'm sure you could automate the decompiling process. The reality is that this technique is security by obscurity at its core, and is therefore never going to succeed.

Skype is probably one example where it took developers 10+ years to figure out how the app worked.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#128

Post by simplyluke »

jazoom wrote: post #125
piker wrote: post #123

Proposition: 99% of scraping use cases are eliminated if the scraper agrees to subsequently abide by the target's terms of service.

Googlebot doesn't abide by 99% of websites' terms of use.

But that's "different" because they've built a $600bn company off it.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#129

Post by skyisblue »

chatmasta wrote: post #52

Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.

Reminds me of when app developers found the private API of the Pokemon GO app and used it to create their own Pokemon locater apps. Many of them made it to the top free charts and would have raked in hundreds of thousands of downloads, if not millions.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#130

Post by jazoom »

simplyluke wrote: post #128
jazoom wrote: post #125

Earlier quoted context omitted.

Googlebot doesn't abide by 99% of websites' terms of use.

But that's "different" because they've built a $600bn company off it.

More that the websites actually want to be found by someone.
Post reply on HN