21–30 of 176 posts
when i need to scrap a webpage, i use phpQuery ( http://code.google.com/p/phpquery/ ), it's dead simple if you have experience with jQuery and i get all the benefits of a server-side programming language.
Great read! In the past, I have successfully used HtmlUnit to fulfill my admittedly limited scraping needs. It runs headless, but it has a virtual head designed to pretend it's a user visting a web application to be be tested for QA purposes. You just program it to go through the motions of a human visting a site to be tested (or scraped). E.g., click here, get some response. For each whatever in the response, click…
I'm surprised that no one has attempted to write a Twitter client based solely on scraping to get around the token limits.
scraping for fun is okay, but if you would like to build a business most people would prefer to abide by the terms of use and still use the API. remember the story of Pad Mapper ?
I've done a ton of scraping (mostly legal: on behalf of end users of an app on sites they have legit access to). This article misses something that affects several sites: JavaScript driven content. Faking headers and even setting cookies doesn't get around this. This is of course is easy to get around, using something like phantom.js or Selenium. Selenium is great because unlike all the whiz bang scraping techniques,…
My 2c about scraping - when you try to obtain data from large websites, always go for javascript content. Pages like Newegg or Amazon * may change html outline very often even without a single alteration to the front-user and even your smartest regex can have a brain fart. In contrast, even when site gets major overhaul, most likely old javascript will be left in place with all up to date variables, because engineers will be concerned of removing that code not to break some functionality .
* given you have rights to scrap.
not that there are no tools to debug the site; but I found websites like mentioned plus youtube, and bunch others just not fiddling too much with js.
Earlier quoted context omitted.
Why? I would gladly encourage web scraping.
Totally agree, scraping is great. I can see site operators being against the practice though, as it (usually): - generates no ad revenue - often enables someone else to use data that you struggled to put together, allowing others to profit with no gain for you - hit's edge cases that were never optimized for (as it does not follow real user access)
The reality of scraping was really known many years ago. If you're doing if for above-board reasons like for research etc., you'll probably get a pass - if you're doing it in order to profit from someone else's work because you are too lazy to do it yourself, it's probably unethical and you won't get a pass --- these concepts have been around for at least a thousand years or more.
Full Disclosure: I have also scraped data - but only from government websites where the scraped data is explicitly public domain to begin with and APIs were not available.
I love HTML scraping. But Javascript???...The juiciest data sets these days are increasingly in JS. For the love of me i can't get around scraping JS :( I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share?? I would be eternally grateful.
first install Xvfb and pyvirtualdisplay then try this snippet https://gist.github.com/4243582
selenium is great, it can even wait for ajax requests to finish (see WebDriverWait) ..
EDIT: Knowledge isn't zero-sum. Here's an overview of a kick-ass way to spider/scrape:
I use Scrapy to spider asynchronously. When I define the crawler bot as an object, if the site contains complicated stuff (stateful forms or javascript) I usually create methods that involve importing either Mechanize or QtWebKit. Xpath selectors are also useful for the ability to not have to specify the entire XML tree from trunk to leaf. I then import pre-existing Django models from a site I want the data to go into and write to the DB. At this point you usually have to convert some types.
I find Scrapy cleaner and more like a pipeline so it seems to produce less 'side effect kludge' than other scraping methods (if anybody has seen a complex Beautiful Soup + Mechanize scraper you know what I mean by 'side effect kludge'). It can also act as a server to return json.
Being asynchronous, you can do crazy req/s.
I will leave out how to do all this through Tor because I don't want the Tor network being abused but am happy to talk about it one on one if your interest is beyond spamming the web.
Through this + a couple of unmentioned tricks, it's possible to get insane data, so much so it crosses over into security research & could be used for pen-testing.
Another issue not covered: file downloads. Let's say you have a process that creates a dynamic image, or logs in and downloads dynamic PDFs. Even Selenium can't handle this (the download dialog is an OS-level feature). At one point I was able to get Chrome to auto-download in Selenium, but had zero control over filename and where it was saving. I ended up using iMacros (the pay version) to drive this (using Windows i…
The issue with web scraping is that it relies on the scraper to keep up with changes made to the site. If a site owner changes the layout or implements a new feature, the programs depending on the scraper immediately fail. This is much less likely to happen when working with official APIs.
This should be stressed - sites like Facebook do exactly this. Constant changes mean constantly updating your scraper. When it comes to A/B testing? Your scraper needs to intelligent find the data, which might not always be in the same place. Sidenote: I wonder if any webapps use randomly generated IDs and class names (linked in the CSS) to prevent scraping. I guess this would be a caching nightmare, though.
At least it's easier to code these tricks than to patch a scraper to get around them.