Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Don’t Need No Stinking API: Web Scraping For Fun and Profit

blog.hartleybrody.com

Post reply on HN
Print view View original

121–130 of 176 posts

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#121

Post by hartleybrody »

kanzure wrote: post #108
kaffeinecoma wrote: post #102

From the article: Since the third party service conducted rate-limiting based on IP address (stated in their docs), my solution was to put the code that hit their service into some client-side Javascript, and then send the results back to my server from each of the clients. This way, the requests would appear to come from thousands of different places, since each client would presumably have their own unique IP addre…

> Since the third party service conducted rate-limiting based on IP By the way, that's one of my projects. You can use a basic fibonacci-related algorithm to figure out (in the most minimal number of requests) what exactly the rate limit is. This way, you can scrape at just under the maximum limit. I am still working on this core library though. :|

Sounds pretty interesting! Be sure to share it when it's ready.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#122

Post by dickbasedregex »

goostavos wrote: post #41
bdcravens wrote: post #16

I've done a ton of scraping (mostly legal: on behalf of end users of an app on sites they have legit access to). This article misses something that affects several sites: JavaScript driven content. Faking headers and even setting cookies doesn't get around this. This is of course is easy to get around, using something like phantom.js or Selenium. Selenium is great because unlike all the whiz bang scraping techniques,…

In regards to the javascript problem. I'd suggest checking out the mobile versions of the sites first before you hop to a weighty solution like Selenium. Could be a very simple solution to the problem :) I recently built a twitter bot that did some scraping and posting. I beat my head against a wall for a couple of hours trying to find a good tool to deal with all the javascript driven stuff. I happened to get an upd…

If you're using ruby, I've found watir (http://watir.com/) to be very nice to use. There might be better alternatives now but it made my life easier when I had to scrape a bunch of our supplier's crappy B2B sites that required JavaScript.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#123

Post by hartleybrody »

kaffeinecoma wrote: post #102

From the article: Since the third party service conducted rate-limiting based on IP address (stated in their docs), my solution was to put the code that hit their service into some client-side Javascript, and then send the results back to my server from each of the clients. This way, the requests would appear to come from thousands of different places, since each client would presumably have their own unique IP addre…

That's a great point, for most web services, this request would be blocked at the browser level by the Same Origin Policy. Fortunately for me, this site allowed client-side calls by returning a Access-Control-Allow-Origin: * header[1], specifically designed to allow this type of cross-domain access.

[1]: http://en.wikipedia.org/wiki/Same_origin_policy#Cross-Origin...

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#125

Post by hartleybrody »

zarino wrote: post #105

Related: If you fancy writing scrapers for fun and profit, ScraperWiki (a Liverpool, UK-based data startup) is currently hiring full-time data scientists. Check us out! http://scraperwiki.com/jobs/#swjob5

very well played :)

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#126

Post by hartleybrody »

mylittlepony wrote: post #86
im3w1l wrote: post #82

People seem to wonder how to handle ajax. The answer is HttpFox. It records all http-requests. 1. Start recording 2. Do some action that causes data to be fetched 3. Stop recording. You will find the url, the returned data, and a nice table of get and post-variables. https://addons.mozilla.org/en-us/firefox/addon/httpfox/

Isn't this the same as what the Net tab from Firebug does?

Yah, I don't understand why people make things so complicated once Javascript gets involved. Just inspect the XHR traffic to your browser ("Network" tab in Web Inspector, Firebug, etc) as you update the information on the page. You'll quickly discover what are essentially undocumented APIs returning the data used to generate the page. You don't need to use or even read through the Javascript that's calling them, you just need to figure out what parameters and cookies are being sent, and tweak those as you wish.

You might have to spoof the Referer header so that it thinks the request is still coming from their website.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#127

Post by polyfractal »

robryan wrote: post #44
cloverich wrote: post #34

Earlier quoted context omitted.

> So you're not so hot on the whole search engine thing? They scrape to generate links for users to go to the site . That's quite different than scraping for...any other purpose? So it seems. Would you (anyone) argue otherwise? (genuine curiosity).

They are also using title, description, some snippets from the page and taking a cached version of the site and images you can view without having to visit the site itself. They are also using this data as a product to sell advertising against. If there wasn't so much benefit for most of all sites to be in search engine indexes you would thinking at least some would object to this scraping. There would be lots of oth…

Google is even moving into the territory of scraping content to display. Relevant wikipedia snippets are now being displayed on the search page as a side bar. While Wiki probably doesn't care...there are plenty of other sites that would not like Google to scrape the content and display it on the search page.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#128

Post by RBerenguel »

treelovinhippie wrote: post #14

Not every site. There is data I would really love to access on Facebook without having to gain specific authorization from the user. It's odd that for most user profiles the most you can extract via the graph API (with no access token) is their name and sex. Whereas I can visit their profile page in the browser, see all sorts of info and latest updates (and not even be friends with them) Tried scraping Facebook. They…

Do it in JS, client-side with 3 second delays. I have used this to get available data (location, name, status, etc) from the latest 500 available likes of a page I manage

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#129

Post by chill1 »

kybernetyk wrote: post #81
notimetorelax wrote: post #77

Earlier quoted context omitted.

Another proof that spending 15 minutes on research can save you days in development and production.

And your comment is another proof that people tend to assume everyone else is an idiot.

I think his comment was quite appropriate, and did not feel there was an implication that he thought anyone was dumb for not having known about the alternative approach to getting Reddit data.

Often times programmers and the managers that drive them are way too quick to get going building or solving something with brute force. If they would just be patient and stop for a moment. Spending even a mere 30 minutes extra doing your homework on a problem can save hours or days in dev time.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#130

Post by shocks »

chadscira wrote: post #107
shocks wrote: post #93

Node.js is excellent for web scaping, especially if you're scraping large amounts very often.

I made this module for this exact reason: https://github.com/icodeforlove/node-requester . Supports horrible things like proxy rotation.

This looks great. I think I will incorporate this into some of my projects.
Post reply on HN