Show HN: BestSFBooks, Mashup of the Best SF/Fantasy Books • Hacker News Board

Re: Show HN: BestSFBooks, Mashup of the Best SF/Fantasy Books

#21

Post by berntb » Wed, Oct 05, 2011, 9:32 PM UTC

bgraves wrote: post #14

gurgeous wrote: post #9

Earlier quoted context omitted.

I've written a lot of data tools as part of Urbanspoon and subsequent startups. I like to collect publicly available data, clean it up, normalize it, and then release it in a more useful way. Hot tips for crawling data: - Cache pages locally while you work on the indexing - Nokogiri is awesome - Don't be afraid to use regular expressions - Initially, put data into a spreadsheet (not the db). That way it can be checke…

Thanks! I've been doing some scraping projects lately and really like it a lot. There's a pretty steep learning curve, but it gets easier and easier as you go along, I think. 1. Caching pages is definitely a great idea while debugging. Especially if the data source has a request limit :) 2. I've never heard of Nokogiri, but it looks like BeautifulSoup for Ruby. I've found that Python has worked for everything I need…

>>I suck so bad at regex

Then you are the opposite of where I used to be -- I thought I understood and could use regexps. Hell, I do Perl for fun. :-)

Something like when I first sat down with Photoshop -- "Hey, I know how to program Macs [before MacOs X]. This is just using a Macintosh program, so I should have no problems"... :-)

Read "Mastering Regular Expressions". It made me feel embarrassed about my previous stupidity [Edit: Embarrassment, your name is Dunning–Kruger :-) ]. Just the first few chapters are enough to change your world.

Edit: I might add, I still can't use Photoshop.

Re: Show HN: BestSFBooks, Mashup of the Best SF/Fantasy Books

#22

Post by pasbesoin » Wed, Oct 05, 2011, 9:34 PM UTC

Yay!

http://www.bestsfbooks.com/b/3119/The-Children-of-the-Sky

The Children of the Sky Series: Zones of Thought #3 by Vernor Vinge (Tor, Oct-2011)

Re: Show HN: BestSFBooks, Mashup of the Best SF/Fantasy Books

#23

Post by dougws » Wed, Oct 05, 2011, 10:01 PM UTC

There seems to be a really heavy bias towards relatively new SF; is this because there are more awards now than there used to be, or because you only have data going so far back?

Re: Show HN: BestSFBooks, Mashup of the Best SF/Fantasy Books

#24

Post by phzbOx » Wed, Oct 05, 2011, 10:04 PM UTC

gurgeous wrote: post #9

bgraves wrote: post #6

"As always, data acquisition and cleanup was the hardest part." I'm most interested in this piece of the project. What were your particular tools and methodologies? How long did it take you, once you identified your data sources? Any interesting stumbling blocks or problems that were solved along the way?

I've written a lot of data tools as part of Urbanspoon and subsequent startups. I like to collect publicly available data, clean it up, normalize it, and then release it in a more useful way. Hot tips for crawling data: - Cache pages locally while you work on the indexing - Nokogiri is awesome - Don't be afraid to use regular expressions - Initially, put data into a spreadsheet (not the db). That way it can be checke…

May I add: Usually, scraping is a "one time only job", so feel free to use all the hack you want to get the job faster. For instance, wget/grep into a file, use vim to clean it up a bit, mix some awk, perl or bash. The goal is just to get the data, not to write production quality code that will be maintained for years.

For tricky pages, it can be a good idea to write tests though. Be sure to cache all downloaded pages so the tests can be run uber-fast.. It's a good way to perfect that regex. (Yes, there are programs for that, but sometime HTML have some weird newlines or characters that screw everything).

Also, I think it's important to emphasis not to "code everything". Say you've got 10 links to get on the first page.. and in all of these links you've got hundreds. Obviously, you won't go on each page and manually copy everything; however, for the first 10 links, it's useless to write a script that crawl that. Just copy it and clean it from the source. Jee, use a macro in emacs or your favorite editor if you really cant repeat a task 10 times.

Re: Show HN: BestSFBooks, Mashup of the Best SF/Fantasy Books

#25

Post by semanticist » Thu, Oct 06, 2011, 1:11 AM UTC

bgraves wrote: post #14

gurgeous wrote: post #9

Earlier quoted context omitted.

I've written a lot of data tools as part of Urbanspoon and subsequent startups. I like to collect publicly available data, clean it up, normalize it, and then release it in a more useful way. Hot tips for crawling data: - Cache pages locally while you work on the indexing - Nokogiri is awesome - Don't be afraid to use regular expressions - Initially, put data into a spreadsheet (not the db). That way it can be checke…

Thanks! I've been doing some scraping projects lately and really like it a lot. There's a pretty steep learning curve, but it gets easier and easier as you go along, I think. 1. Caching pages is definitely a great idea while debugging. Especially if the data source has a request limit :) 2. I've never heard of Nokogiri, but it looks like BeautifulSoup for Ruby. I've found that Python has worked for everything I need…

I spent the first half of this year writing scrapers for every newspaper in the UK. My Top Regex Tip is http://rubular.com/ - this thing saved me HOURS of my life.

Re: Show HN: BestSFBooks, Mashup of the Best SF/Fantasy Books

#26

Post by jogrimst » Fri, Oct 07, 2011, 5:46 PM UTC

gigawatt wrote: post #19

This is fantastic! Perfectly simple and useful. Only glitch I've run into so far is that the search function doesn't work. As far as I can tell, there's no type="submit" for the form.

Hello. That is because it is a auto-complete search field. If you search for something that does not exist, nothing will happen. If you type for example "Krak", you will get auto-complete suggestion for "Kraken".

Maybe it would be an idea to show a message like "No results..." if what the user types does not return any matches in this search function?