R3, a map-reduce engine with Python and Redis
heynemann.github.com
R3, a map-reduce engine with Python and Redis
1–10 of 40 posts
Re: R3, a map-reduce engine with Python and Redis
#2This looks like an interesting project.
Is there something like this for php?
Re: R3, a map-reduce engine with Python and Redis
#3Anyone have some insight into situations where running map reduce on redis makes more sense than other software like the traditional hadoop?
Re: R3, a map-reduce engine with Python and Redis
#4Anyone have some insight into situations where running map reduce on redis makes more sense than other software like the traditional hadoop?
Hadoop is a bloated pile of elephant poo. Any and all alternatives are welcome. Disco (http://discoproject.org/) is popular in some parts of the mapreducesphere.
Re: R3, a map-reduce engine with Python and Redis
#5I can think of one case where a redis dictionary is used to represent a tree, and reductions are needed over a subtree. Calculations on river networks are like this. You might want to use redis instead of a cPickled dictionary, and you might not want the overhead of a full Hadoop.
Re: R3, a map-reduce engine with Python and Redis
#6Anyone have some insight into situations where running map reduce on redis makes more sense than other software like the traditional hadoop?
Hadoop is a bloated pile of elephant poo. Any and all alternatives are welcome. Disco ( http://discoproject.org/ ) is popular in some parts of the mapreducesphere.
I hear the above comment about Hadoop a lot. Can you explain why?
Re: R3, a map-reduce engine with Python and Redis
#7This looks like an interesting project. Is there something like this for php?
Python isn't a hard language to learn. It's probably easier to learn Python than to port this to PHP.
Re: R3, a map-reduce engine with Python and Redis
#8Anyone have some insight into situations where running map reduce on redis makes more sense than other software like the traditional hadoop?
Hadoop is a bloated pile of elephant poo. Any and all alternatives are welcome. Disco ( http://discoproject.org/ ) is popular in some parts of the mapreducesphere.
Using disco here, very happy with it.
Re: R3, a map-reduce engine with Python and Redis
#9Earlier quoted context omitted.
Hadoop is a bloated pile of elephant poo. Any and all alternatives are welcome. Disco ( http://discoproject.org/ ) is popular in some parts of the mapreducesphere.
I hear the above comment about Hadoop a lot. Can you explain why?
I'd love to, but it would take about an hour to run through everything.
Here's a short version: There's a collective ecosystem problem of fragmented applications, not-quite-right command line utilities, web interfaces that look like they were designed in 1995, noisy log files people actually have to read constantly, and cross coupling of dependencies that make keeping a cluster live for production use a full time job.
There's the programming problem of nobody actually writing hadoop mapreduce code because it's impossibly complicated. Everybody uses hive and pig and half a dozen other tools to compile to pre-templated java classes (this knocks off 5% to 30% of your performance if you could do it by hand).
It hasn't grown because it's so amazing, performant, and company saving. It grows because people jumped on a fad wagon then got stuck with having a few hundred TB in HDFS. The lack of a competing project with equal mindshare and battle-testedness doesn't foster any competition. It's the mysql of distributed processing systems. It works (mostly), but it breaks (in a few dozen known ways), so people keep adding features and building on top of it.
Re: R3, a map-reduce engine with Python and Redis
#10Can you horizontaly scale the redis backend or it supports only one instance?
Why restrain to sequential reducers when you can parallelize with partitions and sorting?