Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Write your first MapReduce program in 20 minutes

michaelnielsen.org

Post reply on HN
Print view View original

1–10 of 16 posts

Re: Write your first MapReduce program in 20 minutes

#2

Post by wedesoft »

And here's the equivalent Ruby program. In Ruby it is usually 'collect' and 'inject' instead of 'map' and 'reduce'.

    result = Dir.glob('test?.txt').collect do |file_name|
      File.new(file_name, 'r').read.split(' ').collect do |word|
        word.downcase.tr '.,\'', ''
      end.inject Hash.new(0) do |hash,word|
        hash[word] += 1
        hash
      end
    end.inject do |all,hash|
      (all.keys + hash.keys).uniq.inject Hash.new(0) do |acc,word|
        acc[word] = all[word] + hash[word]
        acc
      end
    end
    p result
Edit: The Python example in the article is better because it merges hashes in the reduce step which facilitates parallelisation.

Re: Write your first MapReduce program in 20 minutes

#5

Post by cdcarter »

wedesoft wrote: post #2

And here's the equivalent Ruby program. In Ruby it is usually 'collect' and 'inject' instead of 'map' and 'reduce'. result = Dir.glob('test?.txt').collect do |file_name| File.new(file_name, 'r').read.split(' ').collect do |word| word.downcase.tr '.,\'', '' end.inject Hash.new(0) do |hash,word| hash[word] += 1 hash end end.inject do |all,hash| (all.keys + hash.keys).uniq.inject Hash.new(0) do |acc,word| acc[word] = al…

You can also use Enumerable#map and Enumerable#reduce, to use names that match the pattern (and #map is arguably more idiomatic than #collect).

Re: Write your first MapReduce program in 20 minutes

#6

Post by klaruz »

cdcarter wrote: post #4

If you really want to try and play with MapReduce on a dataset without going through setting up a Hadoop node (or 4), check out CouchDB. It's designed around MapReduces (though not distributed), and you even get to deal with solving re-reduce problems.

BigCouch will let you build indexes in parallel.

Re: Write your first MapReduce program in 20 minutes

#7

Post by chrismealy »

cdcarter wrote: post #5
wedesoft wrote: post #2

And here's the equivalent Ruby program. In Ruby it is usually 'collect' and 'inject' instead of 'map' and 'reduce'. result = Dir.glob('test?.txt').collect do |file_name| File.new(file_name, 'r').read.split(' ').collect do |word| word.downcase.tr '.,\'', '' end.inject Hash.new(0) do |hash,word| hash[word] += 1 hash end end.inject do |all,hash| (all.keys + hash.keys).uniq.inject Hash.new(0) do |acc,word| acc[word] = al…

You can also use Enumerable#map and Enumerable#reduce, to use names that match the pattern (and #map is arguably more idiomatic than #collect).

I finally got inject() when they added the reduce() alias.

Re: Write your first MapReduce program in 20 minutes

#8

Post by pangram »

I wrote a little shim that allowed to write me to write Hadoop jobs in Clojure, and had two small test functions that would apply a map / reduce to a test file -- it made development of Hadoop jobs a bit easier. See: https://github.com/brool/hadoop-shim/blob/master/wordcount.c...

Re: Write your first MapReduce program in 20 minutes

#9

Post by [deleted] »

wedesoft wrote: post #2

And here's the equivalent Ruby program. In Ruby it is usually 'collect' and 'inject' instead of 'map' and 'reduce'. result = Dir.glob('test?.txt').collect do |file_name| File.new(file_name, 'r').read.split(' ').collect do |word| word.downcase.tr '.,\'', '' end.inject Hash.new(0) do |hash,word| hash[word] += 1 hash end end.inject do |all,hash| (all.keys + hash.keys).uniq.inject Hash.new(0) do |acc,word| acc[word] = al…

[deleted]

Re: Write your first MapReduce program in 20 minutes

#10

Post by cdcarter »

chrismealy wrote: post #7
cdcarter wrote: post #5

Earlier quoted context omitted.

You can also use Enumerable#map and Enumerable#reduce, to use names that match the pattern (and #map is arguably more idiomatic than #collect).

I finally got inject() when they added the reduce() alias.

The thing that confused me most about #inject was the term 'memo'...
Post reply on HN