Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Don’t Need No Stinking API: Web Scraping For Fun and Profit

blog.hartleybrody.com

Post reply on HN
Print view View original

161–170 of 176 posts

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#161

Post by mcgwiz »

It's pointless to think of it as "wrong" for third-parties to web-scrape. Entities will do as they must to survive. The onus of mitigating web scraping, if in the interests of the publisher, is on the publisher.

As a startup developer, third-party scraping is something I need to be aware of, that I need to defend against if doing so suits my interests. A little bit of research shows that this is not impractical. Dynamic IP restrictions (or slowbanning), rudimentary data watermarking, caching of anonymous request output all mitigate this. Spot-checking popular content by running it through Google Search requires all of five minutes per week. At that point, the specific situation can be addressed holistically (a simple attribution license might make everyone happy). With enough research, one might consider hellbanning the offender (serving bogus content to requests satisfying some certain heuristic) as a deterrent. A legal pursuit with its cost would likely be a last resort.

Accept the possibility of being scraped and prepare accordingly.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#162

Post by lazyfunctor »

latchkey wrote: post #7
kevinpfab wrote: post #5

The issue with web scraping is that it relies on the scraper to keep up with changes made to the site. If a site owner changes the layout or implements a new feature, the programs depending on the scraper immediately fail. This is much less likely to happen when working with official APIs.

In most cases, the site doesn't have an api... so we scrape and take the risk that the structure will change. One thing that helps is using tools which give you jquery-like selectors because they give a lot of freedom and are very easy to write/update.

I agree, CSS selectors in BeautifulSoup and pyquery make it less messy.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#163

Post by rsingel »

hartleybrody wrote: post #157
rsingel wrote: post #155

There are some recent federal cases (Weev http://www.wired.com/opinion/2012/11/att-ipad-hacker-when-em... , Aaron Swartz http://www.wired.com/threatlevel/2012/09/aaron-swartz-felony... , and a prosecution of scalpers http://www.wired.com/threatlevel/2010/07/ticketmaster/ ) that view scraping as a felony hacking offense. The feds think that an attempt to evade CAPTCHAS, IP and MAC blocks is a felony worthy of years in…

Very good point, I've added the following disclaimer: While scraping can sometimes be used as a legitimate way to access all kinds of data on the internet, it’s also important to consider the legal implications. As was pointed out in the comments on HN[1], there are many cases where scraping data may be considered illegal, or open you to the possibility of being sued. Similar to using a firearm, some uses of web scra…

Thanks for adding that and linking. The feds are nutter butter these days.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#165

Post by thomasrambaud »

I think the author just completly missed the point with API vs Screen scraping. The API allows for accessing structured data. Even if the website changes once, the datas would be accessible the same way through the API. Whereas, the author, would have to rewrite his code each time an update his made to the front-office code of the website.

A simple API providing simple json response with http basic auth is far more efficient than a screen scraping program where you have to parse the response using HTML / XML parsers.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#166

Post by frabcus »

thomasrambaud wrote: post #165

I think the author just completly missed the point with API vs Screen scraping. The API allows for accessing structured data. Even if the website changes once, the datas would be accessible the same way through the API. Whereas, the author, would have to rewrite his code each time an update his made to the front-office code of the website. A simple API providing simple json response with http basic auth is far more e…

This isn't always the case - APIs often change. Facebook, for example, is (at least was, a few years ago) notoriously bad at changing in an unpredictable and buggy way, and I stopped using it for that reason. Some HTML scrapers are more reliable than that.

As for efficiency, again not such an issue. HTML is very good these days, compared to 10 years ago, a simple CSS selector often does the job.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#167

Post by thomasrambaud »

frabcus wrote: post #166
thomasrambaud wrote: post #165

I think the author just completly missed the point with API vs Screen scraping. The API allows for accessing structured data. Even if the website changes once, the datas would be accessible the same way through the API. Whereas, the author, would have to rewrite his code each time an update his made to the front-office code of the website. A simple API providing simple json response with http basic auth is far more e…

This isn't always the case - APIs often change. Facebook, for example, is (at least was, a few years ago) notoriously bad at changing in an unpredictable and buggy way, and I stopped using it for that reason. Some HTML scrapers are more reliable than that. As for efficiency, again not such an issue. HTML is very good these days, compared to 10 years ago, a simple CSS selector often does the job.

This is true, but APIs are often versionned.

Concerning efficiency this is true CSS / XPath processors, at least, both offer very nice performances.

But download 70KB of HTML each time you only need a single data, where the API request cost only a few (avg < 2KB), can be such a pain if you need to do this frequently. This can be handled by a scalable configuration but I find it a bit the overkill.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#168

Post by thomasrambaud »

im3w1l wrote: post #82

People seem to wonder how to handle ajax. The answer is HttpFox. It records all http-requests. 1. Start recording 2. Do some action that causes data to be fetched 3. Stop recording. You will find the url, the returned data, and a nice table of get and post-variables. https://addons.mozilla.org/en-us/firefox/addon/httpfox/

Firebug is a lot better.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#169

Post by notimetorelax »

joesb wrote: post #142
chill1 wrote: post #129

Earlier quoted context omitted.

I think his comment was quite appropriate, and did not feel there was an implication that he thought anyone was dumb for not having known about the alternative approach to getting Reddit data. Often times programmers and the managers that drive them are way too quick to get going building or solving something with brute force. If they would just be patient and stop for a moment. Spending even a mere 30 minutes extra…

Saying that that was a proof of spending 15 minutes to search about Reddit API would save his time implied that kybernetyk didn't do that research. But kybernetyk already said he did the research before and that Reddit's API is not good enough for his requirement. So this is not the case that 15 minutes of research will save the time. And his comment meant he assumed kybernetyk didn't do research, i.e. being dumb for…

Let me tell you what I thought when I wrote it...

I did not assume that kybernetyk was dumb or anything, I simply chuckled and thought to myself ouch haven't I done similar mistake before?! Please don't assume the worst when reading someone's comment.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#170

Post by frabcus »

jonpaul wrote: post #144
freshhawk wrote: post #141

Earlier quoted context omitted.

That's really interesting, thanks. I worry that it's not going to replicate a real browser accurately enough, but I'm excited to try it out a bit.

Your worry is correct. http://news.ycombinator.com/item?id=4896054 I've tried scraping with it, and it failed miserably on some sites.

Yeah, it's not mature enough yet.

We're also trying it for integration tests, as it is much quicker than Phantom or Selenium. Even there, where we control the standards-compliant site, it isn't quite good enough yet.

Would love to see more people helping make it so, though!

Post reply on HN