21–26 of 26 posts
#!/usr/bin/R-2.10/bin/Rscript
...
Do you run this as a cron job, or is there some hook from python?
In case you missed it in 2.dm/2-dm_offline.r #!/usr/bin/R-2.10/bin/Rscript ... You can call R as a script. Wicked! Do you run this as a cron job, or is there some hook from python?
Google App Engine doesn't support R but you can use NumPy:
http://code.google.com/appengine/docs/python/python27/using2...
The hardest thing I find about writing a scraper/miner/api consumer is the HUGE amount of irregular data that you have to check for. I was attempting to write a crawler for a pretty big dynamic site. It worked, but the code ended up being so messy because of all the weird quirks I had to check for non-stop.
Basically, write a whole bunch of rules, fail safes, etc. So for a twitter crawler the tweet should always be there, the username should always be there, and the response should always be more than 400 characters.
Then any response that fails doesn't bring down your app, it just goes into the "needs review pile" and your app continues unless it gets 50 or so of these errors in a row. Then it SMSes or emails you to let you know it shut down.
It isn't like normal programming, it is dirty and any analysis you plan to do has to take the assumption that you will never get all of the data.
In terms of cleaner code, TDD will help here, but yeah, you will need dozens if not hundreds of methods while you're trying to classify a response.
The hardest thing I find about writing a scraper/miner/api consumer is the HUGE amount of irregular data that you have to check for. I was attempting to write a crawler for a pretty big dynamic site. It worked, but the code ended up being so messy because of all the weird quirks I had to check for non-stop.
I would have had to charge my users a lot of money to pay for the kind of scraping required to power my product.
The hardest thing I find about writing a scraper/miner/api consumer is the HUGE amount of irregular data that you have to check for. I was attempting to write a crawler for a pretty big dynamic site. It worked, but the code ended up being so messy because of all the weird quirks I had to check for non-stop.
I've done likewise but actually just found that scraping at scale (even with something like EC2) is very expensive - which sort of precludes having any sort of free tier/freemium model around the resulting site. I would have had to charge my users a lot of money to pay for the kind of scraping required to power my product.