Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Don’t Need No Stinking API: Web Scraping For Fun and Profit

blog.hartleybrody.com

Post reply on HN
Print view View original

111–120 of 176 posts

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#111

Post by kanzure »

level09 wrote: post #27
RaSoJo wrote: post #17

I love HTML scraping. But Javascript???...The juiciest data sets these days are increasingly in JS. For the love of me i can't get around scraping JS :( I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share?? I would be eternally grateful.

That is actually very simple, and you can even use a headless browser to execute javascript: first install Xvfb and pyvirtualdisplay then try this snippet https://gist.github.com/4243582 selenium is great, it can even wait for ajax requests to finish (see WebDriverWait) ..

> first install Xvfb and pyvirtualdisplay

You really don't need xvfb anymore. Use xserver-xorg-video-dummy.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#112

Post by chadscira »

kanzure wrote: post #110
chadscira wrote: post #107

Earlier quoted context omitted.

I made this module for this exact reason: https://github.com/icodeforlove/node-requester . Supports horrible things like proxy rotation.

> Supports horrible things like proxy rotation. Do you have any plans to track which proxies are actually working, or how quickly each one is blocked? I want a reverse proxy on my outgoing requests that knows how to shift my traffic around properly so that I don't get banned. I don't want to be rate limited and I don't want to sit here for weeks trying to figure out wtf the rate limit is in the first place.

interesting, im sure you can build this into it. you could hook onto the didRequestFail method and flag IPs (log this.proxy out to see what the proxy was). all i would need to do is add a method that makes it easier to add/remove proxies.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#113

Post by kanzure »

alexmic wrote: post #88
RaSoJo wrote: post #17

I love HTML scraping. But Javascript???...The juiciest data sets these days are increasingly in JS. For the love of me i can't get around scraping JS :( I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share?? I would be eternally grateful.

If you are using Python, you can also use pyv8 to evaluate Javascript code.

Yes, but if you want the DOM you would have to use something like webkit. So something like pyphantomjs might hit the right spot. It's a python re-implementation of phantomjs.

https://github.com/kanzure/pyphantomjs

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#115

Post by lazyjones »

Scraping could be made a lot harder by website publishers, but they all depend on the biggest scraper accessing their content so it can bring traffic: Google ...

The biggest downside of scraping is that it often takes a long time for very little content (e.g. scraping online stores with extremely bloated HTML and 10-25 products/per page).

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#116

Post by JoeAltmaier »

lazyjones wrote: post #115

Scraping could be made a lot harder by website publishers, but they all depend on the biggest scraper accessing their content so it can bring traffic: Google ... The biggest downside of scraping is that it often takes a long time for very little content (e.g. scraping online stores with extremely bloated HTML and 10-25 products/per page).

As a pioneer of scraping (NetProphet, the first interactive stock charting app with push-data) we initially scraped every quote we had in our database from other sites.

The fundamental problem is, web pages can change a lot. We constantly had scraper scripts fail either because the web pages changed for some innocuous reason, or they noticed the scraping and blocked us.

We resorted to a list of scrape targets and constantly-updating scrape-scripts to adapt continuously to the 'market'. We also pinged each target to find the least congested.

Eventually we got our own stock feed (guy that did that is a research scientist at Adobe now) and stopped scraping altogether. But it was a wild ride.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#117

Post by lazyjones »

mbustamante wrote: post #10

when i need to scrap a webpage, i use phpQuery ( http://code.google.com/p/phpquery/ ), it's dead simple if you have experience with jQuery and i get all the benefits of a server-side programming language.

For Perl you can use Web::Query (but don't use the default HTML::TreeBuilder::XPath with it, it's extremely slow - use http://search.cpan.org/dist/HTML-TreeBuilder-LibXML/), or Mojo::DOM (part of Mojolicious: http://mojolicio.us/). Both use the nowdays standard CSS selectors for comfortable handling.

JavaScript can be scraped using WWW::Mechanize::Plugin::JavaScript or even WWW::Mechanize::Firefox).

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#118

Post by suldan34 »

freshhawk wrote: post #38
RaSoJo wrote: post #17

I love HTML scraping. But Javascript???...The juiciest data sets these days are increasingly in JS. For the love of me i can't get around scraping JS :( I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share?? I would be eternally grateful.

Phantom.js and casper.js If you can't get the data from the endpoints the javascript hits then write your scraper in javascript and have it run in a headless browser, and it's the webkit engine so most sites test their site against it heavily. Either pull the data out of the javascript objects or trigger your extraction from the html by attaching to the events in the javascript.

upvote for casperjs - it's definitely the best system I've come across for scraping javascript / ajax contents.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#119

Post by lazyjones »

JoeAltmaier wrote: post #116
lazyjones wrote: post #115

Scraping could be made a lot harder by website publishers, but they all depend on the biggest scraper accessing their content so it can bring traffic: Google ... The biggest downside of scraping is that it often takes a long time for very little content (e.g. scraping online stores with extremely bloated HTML and 10-25 products/per page).

As a pioneer of scraping (NetProphet, the first interactive stock charting app with push-data) we initially scraped every quote we had in our database from other sites. The fundamental problem is, web pages can change a lot. We constantly had scraper scripts fail either because the web pages changed for some innocuous reason, or they noticed the scraping and blocked us. We resorted to a list of scrape targets and con…

We still need to scrape many (several 100) clients' websites because they are unable to give us product feeds (adequate ones or any at all) for their stores. But hey, it gives us a small edge because we try harder than the competition.
Post reply on HN