I Don’t Need No Stinking API: Web Scraping For Fun and Profit
blog.hartleybrody.com
I Don’t Need No Stinking API: Web Scraping For Fun and Profit
1–10 of 176 posts
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#2An important topic.
The main caveat is that this may violate a site's terms of use and thus website owners may feel called upon to sue you. Depending on circumstances, the legal situation here can be a long story.
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#3I'm surprised that no one has attempted to write a Twitter client based solely on scraping to get around the token limits.
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#4And this is why we can't have nice things.
Web scraping, as fun as it is (and btw, this title again abuses "Fun and Profit"), is not a practice we should encourage. Yes, it's the sort of dirty practice many people do, at one point or another, but it shouldn't be glorified.
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#5The issue with web scraping is that it relies on the scraper to keep up with changes made to the site.
If a site owner changes the layout or implements a new feature, the programs depending on the scraper immediately fail. This is much less likely to happen when working with official APIs.
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#6And this is why we can't have nice things. Web scraping, as fun as it is (and btw, this title again abuses "Fun and Profit"), is not a practice we should encourage. Yes, it's the sort of dirty practice many people do, at one point or another, but it shouldn't be glorified.
Why? I would gladly encourage web scraping.
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#7The issue with web scraping is that it relies on the scraper to keep up with changes made to the site. If a site owner changes the layout or implements a new feature, the programs depending on the scraper immediately fail. This is much less likely to happen when working with official APIs.
In most cases, the site doesn't have an api... so we scrape and take the risk that the structure will change. One thing that helps is using tools which give you jquery-like selectors because they give a lot of freedom and are very easy to write/update.
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#8I'm surprised that no one has attempted to write a Twitter client based solely on scraping to get around the token limits.
Or an alternative API that uses the scraped data from Twitter to make requests... but that might be getting a bit ambitious (and legally dodgy)
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#9And this is why we can't have nice things. Web scraping, as fun as it is (and btw, this title again abuses "Fun and Profit"), is not a practice we should encourage. Yes, it's the sort of dirty practice many people do, at one point or another, but it shouldn't be glorified.
Hmm,
I'm not actually in favor of scraping. But I think it is possibility that needs to be considered on both ends. If a site has valuable info and doesn't provide a decent API, it naturally is going to encourage scraping.
And isn't a search engine a kind of scraper?
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#10when i need to scrap a webpage, i use phpQuery (http://code.google.com/p/phpquery/), it's dead simple if you have experience with jQuery and i get all the benefits of a server-side programming language.