I Don’t Need No Stinking API: Web Scraping For Fun and Profit • Hacker News Board

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#81

Post by kybernetyk » Sun, Dec 09, 2012, 10:55 AM UTC

notimetorelax wrote: post #77

traxtech wrote: post #72

Earlier quoted context omitted.

Reddit JSON api (just add .json to any URL) is not good enough for you ?

Another proof that spending 15 minutes on research can save you days in development and production.

And your comment is another proof that people tend to assume everyone else is an idiot.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#82

Post by im3w1l » Sun, Dec 09, 2012, 10:57 AM UTC

People seem to wonder how to handle ajax.

The answer is HttpFox. It records all http-requests.

1. Start recording

2. Do some action that causes data to be fetched

3. Stop recording.

You will find the url, the returned data, and a nice table of get and post-variables.

https://addons.mozilla.org/en-us/firefox/addon/httpfox/

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#83

Post by clark-kent » Sun, Dec 09, 2012, 11:05 AM UTC

The problem with scraping instead of using the API is that when the website makes even a slight change to their markup it breaks your code. I have had that experience and it's a living hell. I can say it's not worth it to scrap when there is an API available.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#84

Post by thenomad » Sun, Dec 09, 2012, 11:16 AM UTC

I had to do some scraping of a rather Javascript-heavy site last year - I found the entire process was made almost trivial using Ruby and Nokugiri. Particularly relevant for a non-uber-programmer like me, it's simple to use, as well as powerful.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#85

Post by thibauts » Sun, Dec 09, 2012, 11:34 AM UTC

bdcravens wrote: post #16

I've done a ton of scraping (mostly legal: on behalf of end users of an app on sites they have legit access to). This article misses something that affects several sites: JavaScript driven content. Faking headers and even setting cookies doesn't get around this. This is of course is easy to get around, using something like phantom.js or Selenium. Selenium is great because unlike all the whiz bang scraping techniques,…

In my experience, you seldom need a full browser to extract data from javascript-heavy sites. You often can make your way with a little bit of reverse engineering, starting from a traffic capture and looking after parameters you dont understand in the HTML/JS code. Usually, there is nothing hidden. Though, when they're effectively trying to make your life harder with JS, it is easily solved by feeding a JS interpreter (like python spidermonkey) the offending algorithm.

Depending on your use case, headless may be simpler, but it has also many drawbacks that don't show at first, the main being that they're not simple to drive from remote processes as queue-consuming devices.

The article suggests BeautifulSoup as a parsing library for python. If I'm not mistaken BeautifulSoup is not actively maintained anymore, and other cleaner and faster solutions exists, like lxml.html. Ian Bicking made a good article on that topic : http://blog.ianbicking.org/2008/03/30/python-html-parser-per...

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#86

Post by mylittlepony » Sun, Dec 09, 2012, 11:42 AM UTC

im3w1l wrote: post #82

People seem to wonder how to handle ajax. The answer is HttpFox. It records all http-requests. 1. Start recording 2. Do some action that causes data to be fetched 3. Stop recording. You will find the url, the returned data, and a nice table of get and post-variables. https://addons.mozilla.org/en-us/firefox/addon/httpfox/

Isn't this the same as what the Net tab from Firebug does?

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#87

Post by pknerd » Sun, Dec 09, 2012, 11:51 AM UTC

If it's not automated and a fewer times, I will prefer IMacro to perform tasks on my behalf. The best part of it that you can integrate a Db to record your desired data.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#88

Post by alexmic » Sun, Dec 09, 2012, 11:58 AM UTC

RaSoJo wrote: post #17

I love HTML scraping. But Javascript???...The juiciest data sets these days are increasingly in JS. For the love of me i can't get around scraping JS :( I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share?? I would be eternally grateful.

If you are using Python, you can also use pyv8 to evaluate Javascript code.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#89

Post by frabcus » Sun, Dec 09, 2012, 12:03 PM UTC

joe_the_user wrote: post #2

An important topic. The main caveat is that this may violate a site's terms of use and thus website owners may feel called upon to sue you. Depending on circumstances, the legal situation here can be a long story.

Yes, it is complicated. That said, this is partly just because there aren't enough cases - and partly because the law hasn't stabilised (took a century to stabilise after invention of printing press). It isn't clear what rights society should grant yet, for maximising business.

My take on it, from ScraperWiki's point of view: http://blog.scraperwiki.com/2012/04/02/is-scraping-legal/

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#90

Post by frabcus » Sun, Dec 09, 2012, 12:04 PM UTC

freshhawk wrote: post #38

RaSoJo wrote: post #17

I love HTML scraping. But Javascript???...The juiciest data sets these days are increasingly in JS. For the love of me i can't get around scraping JS :( I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share?? I would be eternally grateful.

Phantom.js and casper.js If you can't get the data from the endpoints the javascript hits then write your scraper in javascript and have it run in a headless browser, and it's the webkit engine so most sites test their site against it heavily. Either pull the data out of the javascript objects or trigger your extraction from the html by attaching to the events in the javascript.

I've started playing with zombie.js recently as well - much lighter and faster than the ones that instrument a completely full browser. But has a full Javascript engine.