Ask HN: Does anyone have experience scraping dynamic content?

#1

Post by sid_viswanathan » Fri, Aug 17, 2012, 10:06 PM UTC

http://www.nfl.com/scores

For example, for this URL, there is a section on the right for "Big Play Highlights" and the first one listed is "L.Brown 7-yard TD pass from..."

It looks like this data is being loaded via some kind of AJAX call. Do you have any ideas on how I can scrape this stream of highlights data? I've never tried to scrape any dynamic content in the past.

Ideas?

Re: Ask HN: Does anyone have experience scraping dynamic content?

#2

Post by bartonfink » Fri, Aug 17, 2012, 10:24 PM UTC

You MAY be able to do this with Selenium, although I've never used Selenium to scrape streaming requests. It's going to depend quite a bit on how the page is structured.

I have used Selenium to scrape dynamic content before by waiting for new DOM elements to be populated by AJAX, so I know this sort of thing is possible in a way you couldn't do with wget.

Re: Ask HN: Does anyone have experience scraping dynamic content?

#3

Post by jfaucett » Fri, Aug 17, 2012, 10:30 PM UTC

yea, just look at the http headers, here's the call: http://www.nfl.com/liveupdate/scores/bigPlayVideos.json?rand...

just make your own tstamp for random

Re: Ask HN: Does anyone have experience scraping dynamic content?

#4

Post by AznHisoka » Tue, Aug 21, 2012, 1:31 AM UTC

PhantomJS/CasperJS is what I've used for my current scraping projects. They're headless browsers and imitate a browser session. Just specify a fake user agent like Mozilla, and you're good to go.

Re: Ask HN: Does anyone have experience scraping dynamic content?

#5

Post by bonsai » Tue, Aug 21, 2012, 7:17 AM UTC

HtmlUnit has very good support for javascript. http://htmlunit.sourceforge.net/