Map Reduce: A simple introduction (2010)

This is a good explanation and I would love to share it but the poor spelling and grammar makes it un-shareable.

Let's use a simple data mining example with logs. You want to count the number of users by region visiting your website to understand where your users are coming from.

In your log files you have the IP which you can use to geolocate those users.

These counts can be caluclated in parallel.

We do this by first "mapping" each line in a log file extracting out the IP. The output of map will be a geolocated hash by region.

Reduce's goal is to calculate an answer based on these keys.

If we think about the output of map being the geo location, we can then collect a count by geo located hash of where our users are coming from.

Map: Map the raw input to some key value space

Reduce: Reduce the output of the map to some aggregate result, think of this like the sql aggregation functions

Hope that helps!

Think of the SQL GROUP BY functionality as an example. You have a giant terabyte-scale text file that is a CSV of observations (say the raw US census results). You want to group people by some facet (say zipcode and household income) and compute some complex, arbitrary function for each group.

Your map function scans the rows and outputs the kv pair (zipcode+","+income, csv line). All the csv lines in a group go to the same instance of the reduce function, where you can run any code you like (compute averages, do deep learning, etc). The output is the results of what you want to compute for each group.

This is a pretty simple example, but does demonstrate where the power of mr comes from -- arbitrary functions in the map and reduce functions that are allowed restricted one-way communication from the mapper to the reducer. It also should help you understand the glib "you can implement SQL on mapreduce" comment below, which is what Apache Hive does.

To be honest, I don't think (and I'm going off the author's name) that english is his first language. Personally, I posted this to HN as it succinctly gave me a quick overview of MR.

I try not to judge an article/blog post on the quality of the grammar when the ideas they are trying to convey are solid and useful.

There are some exceptionally fantastic minds out there that are able to succinctly explain, sometimes difficult, subjects.

I appreciate them for at least trying, and if they explained it in a language that is not their first language, all the more respect to them.

While I agree that numerous spelling and grammar mistakes can indicate a lack of maturity in writing skills, I also know that some of the most brilliant people I've ever worked with were absolutely terrible at spelling. One guy in particular was an amazing programmer. He produced libraries of code that were fast, efficient, and easy to read and understand. They were also very well documented. I know, because I went through the documentation and fixed the dozens and dozens of spelling errors. If I didn't know otherwise, I'd have thought he made a game out of how poorly he spelled words.

I can appreciate numerous spelling/grammar errors making you analyze an article with a bit more scrutiny. However, I can't think of a single example of an article with those kind of errors where I couldn't figure out from the content whether I thought the author really knew what they were talking about.

At any rate, I think there is a big difference between an article needing a little bit more scrutiny, and the article being "unsharable".