Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Don’t Need No Stinking API: Web Scraping For Fun and Profit

blog.hartleybrody.com

Post reply on HN
Print view View original

61–70 of 176 posts

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#61

Post by shardling »

derrida wrote: post #50
bdcravens wrote: post #47

Earlier quoted context omitted.

I've found that regex is very brittle when you don't control what comes across. DOM traversal is far more reliable.

Agree... basically search methods that specify a branch or leaf locally rather than the entire tree structure can more often resist layout changes. Regex for HTML is a bad idea ... http://stackoverflow.com/questions/590747/using-regular-expr...

Parsing arbitrary HTML is not the same as scraping a page for data -- that link isn't really that relevant.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#62

Post by jswanson »

freshhawk wrote: post #36
rizzom5000 wrote: post #26

Earlier quoted context omitted.

Right, and this is why sites like Craigslist explicitly forbid scraping. If the site operators wanted, explicitly, to share their data with you, they would provide an API or give you permission to scrape. The reality of scraping was really known many years ago. If you're doing if for above-board reasons like for research etc., you'll probably get a pass - if you're doing it in order to profit from someone else's work…

1. That doesn't address search engines, which are doing it to profit from someone else's work. If you open the door for search engines then how many search engine like things do you give passes to? 2. What if I'm scraping it just for me, because I want a different interface? How many friends can I share that with? Can I open source the program? 3. What if I read a bunch of these sites to do research and write up a st…

Search engines:

- Respect robots.txt (as mentioned elsewhere) which will often provide a limited subset of all data available

- Give something in return (potential traffic) for the data they reap.

I fully agree that scraping is great, and do it myself frequently. Site operators do have legitimate concerns in some situations though, and it probably comes from feeling as if they are being 'ripped off' somehow.

No one in their right mind is going to object to incidental scraping for personal use.

However, scraping is often scripted into cron or the like and that data is then used to profit someone else. I'm usually cool with that, but if someone is running a web site and they are dependent upon ad revenue to keep the servers running, I understand objecting to it.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#63

Post by kybernetyk »

goostavos wrote: post #41
bdcravens wrote: post #16

I've done a ton of scraping (mostly legal: on behalf of end users of an app on sites they have legit access to). This article misses something that affects several sites: JavaScript driven content. Faking headers and even setting cookies doesn't get around this. This is of course is easy to get around, using something like phantom.js or Selenium. Selenium is great because unlike all the whiz bang scraping techniques,…

In regards to the javascript problem. I'd suggest checking out the mobile versions of the sites first before you hop to a weighty solution like Selenium. Could be a very simple solution to the problem :) I recently built a twitter bot that did some scraping and posting. I beat my head against a wall for a couple of hours trying to find a good tool to deal with all the javascript driven stuff. I happened to get an upd…

For my twitter bot I extracted the xauth keys from twitter's official Mac client(s) (Tweetie 1 and 2 have different keys) and used those to access the API. To twitter the bot looked like the official client and they couldn't ban it without banning their official clients.

And XAuth made account creation and log in a breeze as there was no need for OAuth tokens - username/password was enough.

But you're not always as lucky as that and many websites are heavily JS driven. For Reddit I had to resort to selenium.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#64

Post by kybernetyk »

codewright wrote: post #43
joering2 wrote: post #25

Earlier quoted context omitted.

+1. Thanks for the hints about Selenium. My 2c about scraping - when you try to obtain data from large websites, always go for javascript content. Pages like Newegg or Amazon * may change html outline very often even without a single alteration to the front-user and even your smartest regex can have a brain fart. In contrast, even when site gets major overhaul, most likely old javascript will be left in place with al…

> even your smartest regex can have a brain fart If you're using regex to solve this sort of problem, your code deserves to break, I'm sorry.

This. You usually traverse the DOM. Either you use some XQuery /XPath magic or a library like beautiful soup.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#65

Post by freshhawk »

jswanson wrote: post #62
freshhawk wrote: post #36

Earlier quoted context omitted.

1. That doesn't address search engines, which are doing it to profit from someone else's work. If you open the door for search engines then how many search engine like things do you give passes to? 2. What if I'm scraping it just for me, because I want a different interface? How many friends can I share that with? Can I open source the program? 3. What if I read a bunch of these sites to do research and write up a st…

Search engines: - Respect robots.txt (as mentioned elsewhere) which will often provide a limited subset of all data available - Give something in return (potential traffic) for the data they reap. I fully agree that scraping is great, and do it myself frequently. Site operators do have legitimate concerns in some situations though, and it probably comes from feeling as if they are being 'ripped off' somehow. No one i…

Good rules of thumb.

> No one in their right mind is going to object to incidental scraping for personal use.

It would almost certainly involve stripping ads when re-purposing the content.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#66

Post by Rayne »

kevinpfab wrote: post #5

The issue with web scraping is that it relies on the scraper to keep up with changes made to the site. If a site owner changes the layout or implements a new feature, the programs depending on the scraper immediately fail. This is much less likely to happen when working with official APIs.

Yeah, the author of the post seemed to imply that web APIs are more likely to change than a website. At least, that's how I took it. Blew my mind.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#67

Post by marcus »

bdcravens wrote: post #16

I've done a ton of scraping (mostly legal: on behalf of end users of an app on sites they have legit access to). This article misses something that affects several sites: JavaScript driven content. Faking headers and even setting cookies doesn't get around this. This is of course is easy to get around, using something like phantom.js or Selenium. Selenium is great because unlike all the whiz bang scraping techniques,…

Not quite ready for prime time but I am working on a project that makes it really easy to grab content from any site using a point and click interface no xpaths selectors or regex.

You enter the url you want to capture data from, it gets loaded in an iframe, you click on the texts you need and set a schedule to receive updates and how(email/twitter dm) that's it.

It supports javascript driven content and can handle practically any website.

http://www.followwww.com

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#68

Post by kybernetyk »

RaSoJo wrote: post #17

I love HTML scraping. But Javascript???...The juiciest data sets these days are increasingly in JS. For the love of me i can't get around scraping JS :( I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share?? I would be eternally grateful.

Here's a little experiment with Reddit-automation using Selenium: https://github.com/jsz/reddit_voting

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#69

Post by NoahSussman »

RaSoJo wrote: post #17

I love HTML scraping. But Javascript???...The juiciest data sets these days are increasingly in JS. For the love of me i can't get around scraping JS :( I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share?? I would be eternally grateful.

In general, any Web functional testing tool can be used as a scraper. Scraping and testing are extremely similar. In both cases, one uses XPath or (hopefully) CSS to locate an element and examine certain aspects of that element's state. A scraper is only different from a functional test in that a scraper is focused only on the state of nodes (potentially) containing human-readable content. That, and a scraper saves the data it collects rather than discarding all data at the end of a test run.

Here's a very old Selenium 1.0 example that scrapes the full, rendered HTML of a page. After performing a scrape like this, I would then feed the HTML into a parser such as Nokogiri http://snipplr.com/view/7906/rendered-wget-with-selenium/

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#70

Post by senthilnayagam »

I have done a bit of scrapping with ruby mechanize, when we hit limits have circumvented by proxy and tor

google as a search engine crawls most all sites, but offers very few usable stuff to other bots

http://www.google.com/robots.txt

Disallow 247 Allow 41

Post reply on HN