Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Don’t Need No Stinking API: Web Scraping For Fun and Profit

blog.hartleybrody.com

Post reply on HN
Print view View original

51–60 of 176 posts

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#51

Post by rizzom5000 »

freshhawk wrote: post #36
rizzom5000 wrote: post #26

Earlier quoted context omitted.

Right, and this is why sites like Craigslist explicitly forbid scraping. If the site operators wanted, explicitly, to share their data with you, they would provide an API or give you permission to scrape. The reality of scraping was really known many years ago. If you're doing if for above-board reasons like for research etc., you'll probably get a pass - if you're doing it in order to profit from someone else's work…

1. That doesn't address search engines, which are doing it to profit from someone else's work. If you open the door for search engines then how many search engine like things do you give passes to? 2. What if I'm scraping it just for me, because I want a different interface? How many friends can I share that with? Can I open source the program? 3. What if I read a bunch of these sites to do research and write up a st…

Good points, but... 1. I do think it addresses search engines because site operators do explicitly give search engines permission to scrape their sites via something called "robots.txt" files otherwise known as the "robots exclusion standard". 2. Like all other scenarios, this one is also likely between you and the site operator. Are you breaking the site's TOU? The answer to that question might help. If you are asking me for the answer to a moral dilemma, I might suggest that you try Shakespeare for some relevant insight to your question(s). 3. See (2). I believe you are incorrect in your last sentence, and in a number of ways, but feel free to disagree.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#52

Post by bdcravens »

goostavos wrote: post #49
bdcravens wrote: post #40

Earlier quoted context omitted.

Awesome. I'd love some hints or links, as I'm always looking to refactor.

In general, if you're going the mechanize route, .retrieve() is the function your looking for. e.g. br = mechanize.Browser() br.retrieve("https://www.google.com/images/srpr/logo3w.png, google_logo.png)[0] Mechanize doesn't really have a proper doc, but just about everything you'd need could be figured out from the very lengthy examples page on their site.

Playing with it now, and while it seems to hit my download need, I can't seem to get it to play nice with sites that are JavaScript dependent. Am I missing something, or is there a way to plugin an underlying WebKit engine?

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#53

Post by ohwp »

cynwoody wrote: post #18
kevinpfab wrote: post #5

The issue with web scraping is that it relies on the scraper to keep up with changes made to the site. If a site owner changes the layout or implements a new feature, the programs depending on the scraper immediately fail. This is much less likely to happen when working with official APIs.

>The issue with web scraping is that it relies on the scraper to keep up with changes made to the site. The OP addresses that point. His contention is, there's a lot more pressure on the typical enterprise to keep their public-facing website in tip-top shape than there is to make sure whatever API they've defined is continuing to deliver results properly. Of course, part of the art of (and fun of) scraping is to see…

Of course, part of the art of (and fun of) scraping is to see if you can make your scraper robust against mere cosmetic changes to the site.

I once had to maintain a (legal) scraper and I can tell you there is no fun in making your scraper robust when the website maintainers are doing there best to keep you from scraping there site. I've seen random class-names and identifiers, switching of DIVs and SPANs (block display). Adding and removing SPANs for nesting/un-nesting elements. And so on. Ofcourse the site likes to keep the SEO, but most of the time it's easy to keep parts out of context for a scraper.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#54

Post by jQueryIsAwesome »

splatzone wrote: post #8
mmastrac wrote: post #3

I'm surprised that no one has attempted to write a Twitter client based solely on scraping to get around the token limits.

Or an alternative API that uses the scraped data from Twitter to make requests... but that might be getting a bit ambitious (and legally dodgy)

Create an script that scraps proxies and then use those to scrap twitter, use it in a Russian domain, claim 140 characters can't be copyrighted, claim that the tweets are being extracted from third party sites that use the twitter API but lack any kind of TOS and disclaimer; sell API access, profit!

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#55

Post by freshhawk »

rizzom5000 wrote: post #51
freshhawk wrote: post #36

Earlier quoted context omitted.

1. That doesn't address search engines, which are doing it to profit from someone else's work. If you open the door for search engines then how many search engine like things do you give passes to? 2. What if I'm scraping it just for me, because I want a different interface? How many friends can I share that with? Can I open source the program? 3. What if I read a bunch of these sites to do research and write up a st…

Good points, but... 1. I do think it addresses search engines because site operators do explicitly give search engines permission to scrape their sites via something called "robots.txt" files otherwise known as the "robots exclusion standard". 2. Like all other scenarios, this one is also likely between you and the site operator. Are you breaking the site's TOU? The answer to that question might help. If you are aski…

Good point. I meant to include a mention of "robots.txt" but I forgot or delete it editing. That's the motivation for number 3. A "robots.txt is the law" philosophy makes some sense to me, but number 3 is an example of a time when I think it falls down. I don't see a distinction between scripting my daily bookmark visits and manually doing it as a meaningful one. What about extensive browser plugins?

This isn't settled legally certainly and it certainly doesn't seem like this is settled ethically either considering the various insane statements that occur when politicians comment on the subject.

Some examples of the specific concepts from a thousand years ago that apply and answer these questions would help me see what you see. I know the basic rules for music sampling and referencing other works when writing and where the line for plagiarism is drawn and the rights for using photography. Don't know the rules for accessing network resources that are open or for using their data.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#57

Post by RaSoJo »

level09 wrote: post #27
RaSoJo wrote: post #17

I love HTML scraping. But Javascript???...The juiciest data sets these days are increasingly in JS. For the love of me i can't get around scraping JS :( I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share?? I would be eternally grateful.

That is actually very simple, and you can even use a headless browser to execute javascript: first install Xvfb and pyvirtualdisplay then try this snippet https://gist.github.com/4243582 selenium is great, it can even wait for ajax requests to finish (see WebDriverWait) ..

Yippie...that worked. Thanks a ton for sharing that code snippet. Am unfortunately more of an analyst. Have only just started picking up coding for data analysis. JS scraping was one area that I always had difficulty with. Not anymore :D

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#58

Post by randomdata »

rizzom5000 wrote: post #51
freshhawk wrote: post #36

Earlier quoted context omitted.

1. That doesn't address search engines, which are doing it to profit from someone else's work. If you open the door for search engines then how many search engine like things do you give passes to? 2. What if I'm scraping it just for me, because I want a different interface? How many friends can I share that with? Can I open source the program? 3. What if I read a bunch of these sites to do research and write up a st…

Good points, but... 1. I do think it addresses search engines because site operators do explicitly give search engines permission to scrape their sites via something called "robots.txt" files otherwise known as the "robots exclusion standard". 2. Like all other scenarios, this one is also likely between you and the site operator. Are you breaking the site's TOU? The answer to that question might help. If you are aski…

On #1, you mentioned before that Craigslist disallows scraping, yet unless you are OmniExplorer, it seems scraping is mostly fair game if robots.txt means anything. The robots.txt standard mentions nothing about it being for search engines [2], so there are no special exclusions for search engines specifically.

Additionally, robots.txt is really for automated link traversal, not scrapers in general. If your scraper is initiated by a user, there is no need to follow robots.txt. Not even Google does when the request is user-initiated [3].

From there, the waters just become really murky. Is lynx a scraper because it doesn't render the way most web browsers do? Does it get a pass because it still adheres to web standards? What if a real scraper adheres to web standards? Maybe it is the storage of scraped data that is the issue? What about caches? I could go on, but I'm sure you see what I'm getting at. It's a very complex issue that is not at all understood.

[1] http://www.craigslist.org/robots.txt

[2] http://www.robotstxt.org/robotstxt.html

[3] http://support.google.com/webmasters/bin/answer.py?hl=en&#38...

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#59

Post by freshhawk »

randomdata wrote: post #33
prezjordan wrote: post #12

Earlier quoted context omitted.

This should be stressed - sites like Facebook do exactly this. Constant changes mean constantly updating your scraper. When it comes to A/B testing? Your scraper needs to intelligent find the data, which might not always be in the same place. Sidenote: I wonder if any webapps use randomly generated IDs and class names (linked in the CSS) to prevent scraping. I guess this would be a caching nightmare, though.

I wonder if any webapps use randomly generated IDs and class names (linked in the CSS) to prevent scraping. In my spare time, I've been playing around with "scrapers" (I like to call them web browsers, personally) that don't even look at markup. My first attempt used a short list of heuristics that proved to be eerily successful for what I was after. To the point I could throw random websites with similar content (di…

I would be really interested in knowing which heuristics or machine learning techniques produced decent results. That's if I can't convince you to open source the code. I'm working on the same problem at the moment.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#60

Post by kuhn »

This is a shameless plug but I've created a service that aims to help with a lot of the issues that OP describes such as rate limiting, JS and scaling. It's a bit like Heroku for web scraping and automation. It's still in beta but if anyone is interested then check out http://tubes.io.
Post reply on HN