31–40 of 43 posts
I did almost this exact same thing back in 2005. I had not programmed anything in YEARS and I had to help write part of a system that performed DB queries, screen scraped, file transfers, etc. In a 24 period I tried Java, Perl and Python and in the end Python won by miles and miles. I now use Python daily for all sorts of tasks. I love it.
Your next step on the path to being a hacker is to share your work. Where can we see your "web scraper" in action?
git clone git://gist.github.com/761348.git gist-761348
Earlier quoted context omitted.
I don't know where can I post it? Probably post it to my blog once I've commented it up. Heading out for dinner with the gf so check later! FYI, it looks like a thing of beauty to me but it probably looks like dogshit to the pros. lol.
You will learn a great deal by posting your work for others to see. At first it will feel uncomfortable. You will be concerned that others will mock your work (and they might). But, by the 3rd or 4th time you post your work you will start to feel comfortable with it. The important point is that others will benefit from your experience. Karma.
I feel like 10,000 miles away from that right now.
Your next step on the path to being a hacker is to share your work. Where can we see your "web scraper" in action?
Does this 'URL' help? I think you have to be a member of github to see this. Nice viral mechanism lol. git clone git://gist.github.com/761348.git gist-761348
https://gist.github.com/761348
Fairly idiomatic code too; particularly impressed with the use of XPath.
Are you scraping glogster or are you with glogster? ;-)
The next step is to learn why web scraping is discouraged, and how to use public APIs instead.
Earlier quoted context omitted.
Does this 'URL' help? I think you have to be a member of github to see this. Nice viral mechanism lol. git clone git://gist.github.com/761348.git gist-761348
This one does :-) https://gist.github.com/761348 Fairly idiomatic code too; particularly impressed with the use of XPath. Are you scraping glogster or are you with glogster? ;-)
Yeah, that was what I did when I didn't know anything about programming, too. The next step is to learn why web scraping is discouraged, and how to use public APIs instead.
Congrats. Programming is made out to be a hard thing and for people who don't think logically it is. But to a logically thinking person it's like the greatest game you'll ever discover.
Programming is a harsh mistress, but the rewards are worth it.
Yeah, that was what I did when I didn't know anything about programming, too. The next step is to learn why web scraping is discouraged, and how to use public APIs instead.
Right, just like google uses public api's to access all those sites. Web scraping is responsible for a very large portion of all the traffic on port 80 and the number of sites that have published APIs is extremely small compared to the total.