Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping a Javascript Heavy Website: Keeping Things Simple

tubes.io

Post reply on HN
Print view View original

11–18 of 18 posts

Re: Web Scraping a Javascript Heavy Website: Keeping Things Simple

#11

Post by bdcravens »

I've done a lot of scraping. Some sites use heavy Javascript frameworks that generate session IDs and request IDs that the XHR requests use to "authenticate" the request. In these situations, the amount of work to reverse engineer that workflow is pretty large. In these situations, I lean on headless Selenium. I know there are some lighter solutions, but Selenium offers some distinct advantages:

1) lot of library support, in multiple languages

2) without having to fake UAs, etc, the requests look more like a regular user (all media assets downloaded, normal browser UA, etc)

3) simple clustering: setting up a Selenium grid is very easy, and switching from local instance of Selenium to using the grid requires very little code change (1 line in most cases)

Re: Web Scraping a Javascript Heavy Website: Keeping Things Simple

#12

Post by cynwoody »

rshomali wrote: post #10
cynwoody wrote: post #9

Earlier quoted context omitted.

http://tubes.io/blog/2013/08/29/web-scraping-javascript-heav...

Still the link doesn't work

The original link is now working. He flipped the date in the URL from the 28th to the 29th, then back again.

Re: Web Scraping a Javascript Heavy Website: Keeping Things Simple

#14

Post by PMan74 »

gwu78 wrote: post #7
rgarcia wrote: post #5

I used to use the network tab for stuff like this, but now I almost exclusively use mitmproxy[0]. Once things get sufficiently complicated, the constant scrolling and clicking around in the network tab feels tedious. Plus it's difficult to capture activity if a site has popups or multiple windows. mitmproxy solves these problems and also has a ton more features like replaying requests and saving to files. My ideal to…

mitmproxy sounds like a lot of overhead if all you want is your own raw HTTP traffic. You can get this without Python, and without mitmproxy. Also, I thought mitmproxy was intended for HTTPS. Even in that case, I'm not sure installing Python and mitmproxy is necessary if all you want is to view your own traffic. You can just run your own CA and a proxy that can terminate SSL (e.g., haproxy). Below is a simple, _light…

Wow I thought mitmproxy looked rough until I saw tcpdump/ngrep/awk? They both work of course but neither look especially easy to use.

We've been using http://www.charlesproxy.com/ for years, great tool (cheap albeit not free)

Re: Web Scraping a Javascript Heavy Website: Keeping Things Simple

#16

Post by cynwoody »

bdcravens wrote: post #11

I've done a lot of scraping. Some sites use heavy Javascript frameworks that generate session IDs and request IDs that the XHR requests use to "authenticate" the request. In these situations, the amount of work to reverse engineer that workflow is pretty large. In these situations, I lean on headless Selenium. I know there are some lighter solutions, but Selenium offers some distinct advantages: 1) lot of library sup…

HtmlUnit† is also effective in such cases. HtmlUnit is intended to automate testing of websites. However, the very facilities that enable it to be useful for that purpose also make it useful for scraping.

A few years ago, I wanted to analyze retail store customer feedback data collected by a third party company. The stores were franchises, and the third party was anointed by the franchising company. The data was presented to the user (franchisee store management) via a fancy web site with its own opinion about how the data should be analyzed. My opinion differed. I wanted the data in low-level, RDBMS-friendly form, so that I could recast it every which way (and come back and do it again a new way I thought of). However, such was not forthcoming (big company, little franchisee).

The solution was to make a robot that put the third party company's portal through its paces at the finest granularity, scraping the numbers into a DB as they tediously appeared. The robot was in JRuby††, allowing access to HtmlUnit's functionality without the tedium of Java coding. It was slow, but I didn't care — run it overnight once a month, then run reports off the DB generated.

The coding approach was simple: Pretend you are a user. Access each page, starting with the login page, and do what the user would do. Scrape the interesting numbers as they appear. Append appropriate rows to the DB.

†http://htmlunit.sourceforge.net/

††http://jruby.org/

Re: Web Scraping a Javascript Heavy Website: Keeping Things Simple

#18

Post by gwu78 »

PMan74 wrote: post #14
gwu78 wrote: post #7

Earlier quoted context omitted.

mitmproxy sounds like a lot of overhead if all you want is your own raw HTTP traffic. You can get this without Python, and without mitmproxy. Also, I thought mitmproxy was intended for HTTPS. Even in that case, I'm not sure installing Python and mitmproxy is necessary if all you want is to view your own traffic. You can just run your own CA and a proxy that can terminate SSL (e.g., haproxy). Below is a simple, _light…

Wow I thought mitmproxy looked rough until I saw tcpdump/ngrep/awk? They both work of course but neither look especially easy to use. We've been using http://www.charlesproxy.com/ for years, great tool (cheap albeit not free)

The parent asked for "lightweight". Have you considered the size of the charlesproxy binaries?
Post reply on HN