71–80 of 176 posts
They could be used for computation, but (mostly) aren't.
Earlier quoted context omitted.
In regards to the javascript problem. I'd suggest checking out the mobile versions of the sites first before you hop to a weighty solution like Selenium. Could be a very simple solution to the problem :) I recently built a twitter bot that did some scraping and posting. I beat my head against a wall for a couple of hours trying to find a good tool to deal with all the javascript driven stuff. I happened to get an upd…
For my twitter bot I extracted the xauth keys from twitter's official Mac client(s) (Tweetie 1 and 2 have different keys) and used those to access the API. To twitter the bot looked like the official client and they couldn't ban it without banning their official clients. And XAuth made account creation and log in a breeze as there was no need for OAuth tokens - username/password was enough. But you're not always as l…
Earlier quoted context omitted.
Agree... basically search methods that specify a branch or leaf locally rather than the entire tree structure can more often resist layout changes. Regex for HTML is a bad idea ... http://stackoverflow.com/questions/590747/using-regular-expr...
Parsing arbitrary HTML is not the same as scraping a page for data -- that link isn't really that relevant.
The issue with web scraping is that it relies on the scraper to keep up with changes made to the site. If a site owner changes the layout or implements a new feature, the programs depending on the scraper immediately fail. This is much less likely to happen when working with official APIs.
This should be stressed - sites like Facebook do exactly this. Constant changes mean constantly updating your scraper. When it comes to A/B testing? Your scraper needs to intelligent find the data, which might not always be in the same place. Sidenote: I wonder if any webapps use randomly generated IDs and class names (linked in the CSS) to prevent scraping. I guess this would be a caching nightmare, though.
These guys do a stellar job on the IP addresses: http://www.hidemyass.com/proxy-list -- the good thing is the data is available for an amazing price.
Other sites I have some across will use large images and css sprites to mask price data.
I write a lot of scrapers for fun, rarely profit, just for the buzz
I love HTML scraping. But Javascript???...The juiciest data sets these days are increasingly in JS. For the love of me i can't get around scraping JS :( I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share?? I would be eternally grateful.
Earlier quoted context omitted.
For my twitter bot I extracted the xauth keys from twitter's official Mac client(s) (Tweetie 1 and 2 have different keys) and used those to access the API. To twitter the bot looked like the official client and they couldn't ban it without banning their official clients. And XAuth made account creation and log in a breeze as there was no need for OAuth tokens - username/password was enough. But you're not always as l…
Reddit JSON api (just add .json to any URL) is not good enough for you ?
Earlier quoted context omitted.
For my twitter bot I extracted the xauth keys from twitter's official Mac client(s) (Tweetie 1 and 2 have different keys) and used those to access the API. To twitter the bot looked like the official client and they couldn't ban it without banning their official clients. And XAuth made account creation and log in a breeze as there was no need for OAuth tokens - username/password was enough. But you're not always as l…
Reddit JSON api (just add .json to any URL) is not good enough for you ?