Live data from Hacker News

MapReduce from the basics to the actually useful (in under 30 minutes)

blog.cloudant.com

11–15 of 15 posts

Re: MapReduce from the basics to the actually useful (in under 30 minutes)

#11
post #9

Earlier quoted context omitted.

Hi Aheilbut, apologies if I over-simplified. Perhaps I also undersold scalability -- that's a key point of this type of approach, but non-obvious on such a small data example.

Your people page on your website isn't loading for me in Chrome on OSX, just FYI.

whoops! Will fix.

Re: MapReduce from the basics to the actually useful (in under 30 minutes)

#12
post #6

Am I crazy or was there not a single reduce query in the post? I've been working on map reduce queries for a little while now, aggregating statistics over feeds and stories posted, normalizing over item frequencies, and generally doing simple stuff. I was very excited to see the second half of MapReduce. The reduce queries are always harder to write. If map is addition, reduce is division. I was hoping for examples o…

Plenty of reduce queries, you just didn't need to write the code ;) Good point, though, good material for the next round. I'm open to suggestions if you have an interesting data set or question.

Here's a great example of something that is most natural in mapreduce: http://www.danvk.org/wp/2007-04-06/nebulabrot/

Re: MapReduce from the basics to the actually useful (in under 30 minutes)

#13
post #6

Am I crazy or was there not a single reduce query in the post? I've been working on map reduce queries for a little while now, aggregating statistics over feeds and stories posted, normalizing over item frequencies, and generally doing simple stuff. I was very excited to see the second half of MapReduce. The reduce queries are always harder to write. If map is addition, reduce is division. I was hoping for examples o…

There are reduce views there - both examples 2 and 3 use the built-in "_sum" reduce, which is equivalent to something like:

    function(key, values, rereduce) { return sum(values); }
With regard to more advanced views: you'd be surprised how far you can get with the built-ins (_sum, _count, _stats); I've built a non-trivial data backend (on Cloudant, natch,) using pretty much entirely _sum reduces. Abusing the reduce with complex calculations doesn't seem to be worth it from either a disk space or query performance standpoint.

Re: MapReduce from the basics to the actually useful (in under 30 minutes)

#14
post #8
post #7

Earlier quoted context omitted.

In what way is terminology badly abused? There isn't a single use of the word "cloud" in the entire article and the only "buzzword" terminology used appears to be used accurately and sparingly. If that isn't your contention, why not explain otherwise? How does the abuse/misuse of terminology detract from the quality of the article or make it inaccurate? Further, your argument that the same could be accomplished in "3…

The phrases that irked me were: "I have yet to meet a database that isn’t a key/value store" Dimensions that are "somewhat orthogonal" "It suffices to say that MapReduce is all about giving programmers an efficient way to consume data without needing to know how or where it is actually stored." I don't think that definition suffices, and it misses (or buries in 'efficient') the rather central point that MapReduce is…

I suppose "somewhat orthogonal" is like being "almost pregnant", perhaps the author was just speaking loosely here.

I agree that the examples were toy ones, but it's precisely these simple things like projections that we see folks struggling with when coming from the relational world.

The answer to "can't you just do this in SQL" is always yes you can. Things change, non-relational dbs were around before relational ones and are now returning for a variety of reasons.

I'd like to see some follow on posts that delve into the subtleties of rereduce, another real pain point for new users.

One thing that is not emphasized enough in my mind is the flexibility that a schema-less database such as BigCouch gives you. Consider the trivial schema one would construct to support the 3 or 4 lines of SQL required for these toy examples and then consider how that schema might evolve as needs for different queries change, as the data grows, as different apps with different O-R mapping issues are brought into play and so on.

A schema-less approach does push more of the complexity into the app layer for sure, but it allows the schema to evolve more naturally. After all schema-less doesn't mean the schema really goes away conceptually.

I agree also that Cloudant is doing great things, that's why I joined the team, that and the free coffee :) Thanks for the feedback.

Post reply on HN