Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Don’t Need No Stinking API: Web Scraping For Fun and Profit

blog.hartleybrody.com

Post reply on HN
Print view View original

131–140 of 176 posts

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#131

Post by graue »

thibauts wrote: post #85
bdcravens wrote: post #16

I've done a ton of scraping (mostly legal: on behalf of end users of an app on sites they have legit access to). This article misses something that affects several sites: JavaScript driven content. Faking headers and even setting cookies doesn't get around this. This is of course is easy to get around, using something like phantom.js or Selenium. Selenium is great because unlike all the whiz bang scraping techniques,…

In my experience, you seldom need a full browser to extract data from javascript-heavy sites. You often can make your way with a little bit of reverse engineering, starting from a traffic capture and looking after parameters you dont understand in the HTML/JS code. Usually, there is nothing hidden. Though, when they're effectively trying to make your life harder with JS, it is easily solved by feeding a JS interprete…

BeautifulSoup is in fact still actively maintained. “The current release is Beautiful Soup 4.1.3 (August 20, 2012).”

http://www.crummy.com/software/BeautifulSoup/

I hear it recommended the most among Pythonistas, and it's plenty clean and fast for my use. But if you're skeptical, I'd still look for a more up to date benchmark (or run your own) rather than rely on results from >4 years ago.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#132

Post by wnight »

rizzom5000 wrote: post #51
freshhawk wrote: post #36

Earlier quoted context omitted.

1. That doesn't address search engines, which are doing it to profit from someone else's work. If you open the door for search engines then how many search engine like things do you give passes to? 2. What if I'm scraping it just for me, because I want a different interface? How many friends can I share that with? Can I open source the program? 3. What if I read a bunch of these sites to do research and write up a st…

Good points, but... 1. I do think it addresses search engines because site operators do explicitly give search engines permission to scrape their sites via something called "robots.txt" files otherwise known as the "robots exclusion standard". 2. Like all other scenarios, this one is also likely between you and the site operator. Are you breaking the site's TOU? The answer to that question might help. If you are aski…

If you don't want your data used by others, don't send it to them.

You explicitly give them permission to have it by going out of your way to install a program on a common port, with a common API, and giving it a directory full of documents to distribute, and not using any form of authentication. The way the web works is that answering is equivalent to granting permission to ask and sending a file is tantamount to granting permission. When you receive a file you don't first receive a permissions document, you receive the file - authentication and contractual obligations come first because there is no later. (This is like the tide, you may not like it but that doesn't mean you can change it, especially not with laws.)

You have many ways to check authentication and legally they can be VERY weak, 1-bit passwords are sufficient, but if you don't restrict access it is open - not just because it's the default, but because it's the technical reality: they didn't hack into your computer to get that file, they asked your document server and it gave it to them!

Robots.txt is a suggestion, for the scraper's benefit! It suggests better links. You're allowed to see the rest (the server sends them to you without a password) but you're unlikely to find good content.

If you're afraid of someone examining data you send them, don't send them the data if they ask. Expecting them to not ask, or once they've received it, to not manipulate it in certain ways because you can't then extract a fee for them doing so is controlling and more-over, doomed to fail.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#133

Post by thibauts »

graue wrote: post #131
thibauts wrote: post #85

Earlier quoted context omitted.

In my experience, you seldom need a full browser to extract data from javascript-heavy sites. You often can make your way with a little bit of reverse engineering, starting from a traffic capture and looking after parameters you dont understand in the HTML/JS code. Usually, there is nothing hidden. Though, when they're effectively trying to make your life harder with JS, it is easily solved by feeding a JS interprete…

BeautifulSoup is in fact still actively maintained. “The current release is Beautiful Soup 4.1.3 (August 20, 2012).” http://www.crummy.com/software/BeautifulSoup/ I hear it recommended the most among Pythonistas, and it's plenty clean and fast for my use. But if you're skeptical, I'd still look for a more up to date benchmark (or run your own) rather than rely on results from >4 years ago.

Looks like things have changed since the last time I checked. Thank you for pointing this out. Next time I'll check y facts twice before posting.

Still, lxml being basically a binding to libxml2 the performance comparison of the two libs should still hold. I heard it recommended too, in a python talk about scraping like 1 or 2 (at most) years ago.

BeautifulSoup may still be better for parsing broken documents, though I never had problems with lxml while using it on a very large variety of sites.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#135

Post by traek »

dualogy wrote: post #106
kybernetyk wrote: post #78

Earlier quoted context omitted.

Obviously not - since I would have used it if it was?

Well, what exactly was actually crucially-missing from the json one?

Reddit's API doesn't give you access to child comments past a certain number, so that could have been it.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#137

Post by freshhawk »

rohamg wrote: post #91
freshhawk wrote: post #59

Earlier quoted context omitted.

I would be really interested in knowing which heuristics or machine learning techniques produced decent results. That's if I can't convince you to open source the code. I'm working on the same problem at the moment.

What about something like http:// tubes.io

We're fine with scrapers and scraping infrastructure, although tubes.io is a very interesting idea.

I'm more interested in what I can do to write fewer scrapers since the content is, at a high level, relatively similar. I've just started with experiments writing "generic" scrapers that try and extract the data without depending on markup. It's going to eventually work well enough but to get the error rate down to an acceptable level is going to take a lot of tweaking and trial and error.

There's a few papers on this, but not much out there. That's why I was interested in someone else working on the same problem in a different space.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#138

Post by fudged71 »

polyfractal wrote: post #127
robryan wrote: post #44

Earlier quoted context omitted.

They are also using title, description, some snippets from the page and taking a cached version of the site and images you can view without having to visit the site itself. They are also using this data as a product to sell advertising against. If there wasn't so much benefit for most of all sites to be in search engine indexes you would thinking at least some would object to this scraping. There would be lots of oth…

Google is even moving into the territory of scraping content to display. Relevant wikipedia snippets are now being displayed on the search page as a side bar. While Wiki probably doesn't care...there are plenty of other sites that would not like Google to scrape the content and display it on the search page.

Well, it probably sucks for Wikipedia because users aren't seeing the Jimmy Wales messages everywhere if they find the content through Google.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#139

Post by freshhawk »

thibauts wrote: post #133
graue wrote: post #131

Earlier quoted context omitted.

BeautifulSoup is in fact still actively maintained. “The current release is Beautiful Soup 4.1.3 (August 20, 2012).” http://www.crummy.com/software/BeautifulSoup/ I hear it recommended the most among Pythonistas, and it's plenty clean and fast for my use. But if you're skeptical, I'd still look for a more up to date benchmark (or run your own) rather than rely on results from >4 years ago.

Looks like things have changed since the last time I checked. Thank you for pointing this out. Next time I'll check y facts twice before posting. Still, lxml being basically a binding to libxml2 the performance comparison of the two libs should still hold. I heard it recommended too, in a python talk about scraping like 1 or 2 (at most) years ago. BeautifulSoup may still be better for parsing broken documents, though…

You can use BeautifulSoup with lxml if you like, although I just use the HTMLParser in lxml these days and don't use BeautifulSoup any more. It seems to work a little better, at least for my uses.

http://lxml.de/elementsoup.html

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#140

Post by notimetorelax »

kybernetyk wrote: post #81
notimetorelax wrote: post #77

Earlier quoted context omitted.

Another proof that spending 15 minutes on research can save you days in development and production.

And your comment is another proof that people tend to assume everyone else is an idiot.

No, I didn't meant it. It's a too common mistake to make.
Post reply on HN