Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Don’t Need No Stinking API: Web Scraping For Fun and Profit

blog.hartleybrody.com

Post reply on HN
Print view View original

101–110 of 176 posts

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#101

Post by bryogenic »

bdcravens wrote: post #52
goostavos wrote: post #49

Earlier quoted context omitted.

In general, if you're going the mechanize route, .retrieve() is the function your looking for. e.g. br = mechanize.Browser() br.retrieve("https://www.google.com/images/srpr/logo3w.png, google_logo.png)[0] Mechanize doesn't really have a proper doc, but just about everything you'd need could be figured out from the very lengthy examples page on their site.

Playing with it now, and while it seems to hit my download need, I can't seem to get it to play nice with sites that are JavaScript dependent. Am I missing something, or is there a way to plugin an underlying WebKit engine?

PhantomJS is capable of downloading binary content from js dependent sites but it is a journey to get it working as it is not an out-of-the-box feature. Instead use CasperJS to drive Phantom and get a ton of snazzy features including simple binary downloads. Happy scraping!

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#102

Post by kaffeinecoma »

From the article:

   Since the third party service conducted rate-limiting based on IP
   address (stated in their docs), my solution was to put the code that
   hit their service into some client-side Javascript, and then send
   the results back to my server from each of the clients.

   This way, the requests would appear to come from thousands of
   different places, since each client would presumably have their own
   unique IP address, and none of them would individually be going over
   the rate limit.
Pretty sure the browser Same Original Policy forbids this. Think about it- if this worked, you'd be able to scrape inside corporate firewalls simply by having users visit your website from behind the firewall.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#103

Post by Dylan16807 »

aleprok wrote: post #79

There is just one major trouble with not needing stinking API. You can not POST as a possible client without requiring them to give their password to you, which actually would give you full access to their account instead of limited access with API.

You seem to be talking about a specific site? Which one?

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#106

Post by dualogy »

kybernetyk wrote: post #78
traxtech wrote: post #72

Earlier quoted context omitted.

Reddit JSON api (just add .json to any URL) is not good enough for you ?

Obviously not - since I would have used it if it was?

Well, what exactly was actually crucially-missing from the json one?

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#107

Post by chadscira »

shocks wrote: post #93

Node.js is excellent for web scaping, especially if you're scraping large amounts very often.

I made this module for this exact reason: https://github.com/icodeforlove/node-requester. Supports horrible things like proxy rotation.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#108

Post by kanzure »

kaffeinecoma wrote: post #102

From the article: Since the third party service conducted rate-limiting based on IP address (stated in their docs), my solution was to put the code that hit their service into some client-side Javascript, and then send the results back to my server from each of the clients. This way, the requests would appear to come from thousands of different places, since each client would presumably have their own unique IP addre…

> Since the third party service conducted rate-limiting based on IP

By the way, that's one of my projects. You can use a basic fibonacci-related algorithm to figure out (in the most minimal number of requests) what exactly the rate limit is. This way, you can scrape at just under the maximum limit. I am still working on this core library though. :|

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#109

Post by kanzure »

im3w1l wrote: post #82

People seem to wonder how to handle ajax. The answer is HttpFox. It records all http-requests. 1. Start recording 2. Do some action that causes data to be fetched 3. Stop recording. You will find the url, the returned data, and a nice table of get and post-variables. https://addons.mozilla.org/en-us/firefox/addon/httpfox/

> The answer is HttpFox. It records all http-requests.

http://mitmproxy.org/

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#110

Post by kanzure »

chadscira wrote: post #107
shocks wrote: post #93

Node.js is excellent for web scaping, especially if you're scraping large amounts very often.

I made this module for this exact reason: https://github.com/icodeforlove/node-requester . Supports horrible things like proxy rotation.

> Supports horrible things like proxy rotation.

Do you have any plans to track which proxies are actually working, or how quickly each one is blocked? I want a reverse proxy on my outgoing requests that knows how to shift my traffic around properly so that I don't get banned. I don't want to be rate limited and I don't want to sit here for weeks trying to figure out wtf the rate limit is in the first place.

Post reply on HN