I cringe at the "I wrote a web scraping program to go through the Brickset database". Brickset and Rebrickable both have awesome APIs that can provide all this data in an easily digestible format.
but he wrote a web scraping program for mining the data, bonded it together using wget, launched several aws instances with hadoop, and deployed a mapreduce job for cleaning up the data. It took three weeks and $200 usd, but damn, modern technology is awesome, it would be impossible to do this just 10 years ago. Thanks google, thanks amazon.
seems overkill. still great to live now for doing that kind of analysis.