31–40 of 176 posts
Otherwise, great article! I agree that BeautifulSoup is a great tool for this.
The issue with web scraping is that it relies on the scraper to keep up with changes made to the site. If a site owner changes the layout or implements a new feature, the programs depending on the scraper immediately fail. This is much less likely to happen when working with official APIs.
This should be stressed - sites like Facebook do exactly this. Constant changes mean constantly updating your scraper. When it comes to A/B testing? Your scraper needs to intelligent find the data, which might not always be in the same place. Sidenote: I wonder if any webapps use randomly generated IDs and class names (linked in the CSS) to prevent scraping. I guess this would be a caching nightmare, though.
but who cares, no one can beat Xpath :)
The issue with web scraping is that it relies on the scraper to keep up with changes made to the site. If a site owner changes the layout or implements a new feature, the programs depending on the scraper immediately fail. This is much less likely to happen when working with official APIs.
This should be stressed - sites like Facebook do exactly this. Constant changes mean constantly updating your scraper. When it comes to A/B testing? Your scraper needs to intelligent find the data, which might not always be in the same place. Sidenote: I wonder if any webapps use randomly generated IDs and class names (linked in the CSS) to prevent scraping. I guess this would be a caching nightmare, though.
In my spare time, I've been playing around with "scrapers" (I like to call them web browsers, personally) that don't even look at markup.
My first attempt used a short list of heuristics that proved to be eerily successful for what I was after. To the point I could throw random websites with similar content (discussion sites, like HN), but vastly dissimilar structures, at it and it would return what I expected about, I'd say, 70% of the time in my tests.
After that, I started introducing some machine learning in an attempt to replicate how I determine what blocks are meaningful. My quick prototype showed mix results, but worked well enough that I feel with some tweaking it could be quite powerful. Sadly, I've become busy with other things and haven't had time to revisit it.
With that, swapping variables and similar techniques to thwart crawlers seems like it would be easily circumvented.
And this is why we can't have nice things. Web scraping, as fun as it is (and btw, this title again abuses "Fun and Profit"), is not a practice we should encourage. Yes, it's the sort of dirty practice many people do, at one point or another, but it shouldn't be glorified.
So you're not so hot on the whole search engine thing? The article does slide into the sketchy side (I've always wanted an excuse to do that client side javascript trick too) but I found it more interesting because of that, these aren't secrets. Maybe if I put my "won't somebody please think of the children" hat on I agree that glorifying using trojan code to potentially ddos someones server to get around rate limits…
They scrape to generate links for users to go to the site. That's quite different than scraping for...any other purpose? So it seems. Would you (anyone) argue otherwise? (genuine curiosity).
when i need to scrap a webpage, i use phpQuery ( http://code.google.com/p/phpquery/ ), it's dead simple if you have experience with jQuery and i get all the benefits of a server-side programming language.
A similar module for node.js: https://github.com/mape/node-scraper
Earlier quoted context omitted.
Totally agree, scraping is great. I can see site operators being against the practice though, as it (usually): - generates no ad revenue - often enables someone else to use data that you struggled to put together, allowing others to profit with no gain for you - hit's edge cases that were never optimized for (as it does not follow real user access)
Right, and this is why sites like Craigslist explicitly forbid scraping. If the site operators wanted, explicitly, to share their data with you, they would provide an API or give you permission to scrape. The reality of scraping was really known many years ago. If you're doing if for above-board reasons like for research etc., you'll probably get a pass - if you're doing it in order to profit from someone else's work…
2. What if I'm scraping it just for me, because I want a different interface? How many friends can I share that with? Can I open source the program?
3. What if I read a bunch of these sites to do research and write up a story on something about it? Not plagiarizing, just summarizing and providing analysis on craigslist rental prices? What if I do this every day? What if I automate that process? The data is transformed just as much as if I had read it myself and crunched the numbers myself, I made just as many requests to the site as my browser would have.
Concepts that have been around a thousand years or more are not fully applicable. Like the printing press, some things alter the scarcity equation for ideas and data distribution and ownership. Considering how little we've agreed on about print after 500 years I have some doubts that this is as closed an issue as you say.
Great read! In the past, I have successfully used HtmlUnit to fulfill my admittedly limited scraping needs. It runs headless, but it has a virtual head designed to pretend it's a user visting a web application to be be tested for QA purposes. You just program it to go through the motions of a human visting a site to be tested (or scraped). E.g., click here, get some response. For each whatever in the response, click…
Headless: Xvfb on Linux. (Virtual Framebuffer, let's you run apps that require a GUI) You can use one of the many options that include Webkit (like phantom.js, the capybara-webkit gem, or Selenium if you want a real browser like Firefox to do the work)
I love HTML scraping. But Javascript???...The juiciest data sets these days are increasingly in JS. For the love of me i can't get around scraping JS :( I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share?? I would be eternally grateful.
If you can't get the data from the endpoints the javascript hits then write your scraper in javascript and have it run in a headless browser, and it's the webkit engine so most sites test their site against it heavily.
Either pull the data out of the javascript objects or trigger your extraction from the html by attaching to the events in the javascript.
Another issue not covered: file downloads. Let's say you have a process that creates a dynamic image, or logs in and downloads dynamic PDFs. Even Selenium can't handle this (the download dialog is an OS-level feature). At one point I was able to get Chrome to auto-download in Selenium, but had zero control over filename and where it was saving. I ended up using iMacros (the pay version) to drive this (using Windows i…
I've done this successfully with Ruby Mechanize.