Neat; but seems to missing copyright notices and an explicit license, which means no one can actually use it or redistribute it with their application.
Likely an oversight. Submit a pull request with a BSD-like license file.
R3, a map-reduce engine with Python and Redis
21–30 of 40 posts
Re: R3, a map-reduce engine with Python and Redis
#22Earlier quoted context omitted.
I'd love to, but it would take about an hour to run through everything. Here's a short version: There's a collective ecosystem problem of fragmented applications, not-quite-right command line utilities, web interfaces that look like they were designed in 1995, noisy log files people actually have to read constantly, and cross coupling of dependencies that make keeping a cluster live for production use a full time job…
seiji pretty much nails it. Hadoop seems to have come out of a weird culture. It is a distributed system with a single point of failure (name node) because its designers insisted on avoiding Paxos (distributed systems are too hard so we'll just make a broken-by-design protocol instead). Another example is that a lot of the database code built on top of Hadoop is designed around one Java hashmap per row which really l…
> surely there must be a better way
Re: R3, a map-reduce engine with Python and Redis
#23"Getting one up in your system is beyond the scope of this document." - 67 characters brew install redis redis-server - 31 characters
I actually like when people focus on the case and don't pollute their manuals with such things. If you don't know how to, it's information you can easily obtain elsewhere and you probably got some homework to do anyway.
Re: R3, a map-reduce engine with Python and Redis
#24Earlier quoted context omitted.
Using disco here, very happy with it.
Mind sharing how long you've been using it and how it compares to hadoop in your opinion? I'm very interesting in hearing your experience.
Same here, I'm really interested in hearing about Disco and potential benefits/costs vs. Hadoop.
Re: R3, a map-reduce engine with Python and Redis
#25Do the map-reduce results get put back into redis? I always worry about OOM problems when I'm putting a somewhat unbounded set of things into redis.
Re: R3, a map-reduce engine with Python and Redis
#26I can think of one case where a redis dictionary is used to represent a tree, and reductions are needed over a subtree. Calculations on river networks are like this. You might want to use redis instead of a cPickled dictionary, and you might not want the overhead of a full Hadoop.
On redis 2.6 you can use Lua, reductions over lists could be done directly on the server.
Re: R3, a map-reduce engine with Python and Redis
#27Anyone have some insight into situations where running map reduce on redis makes more sense than other software like the traditional hadoop?
Hadoop is a bloated pile of elephant poo. Any and all alternatives are welcome. Disco ( http://discoproject.org/ ) is popular in some parts of the mapreducesphere.
I <B disco. It is so well designed and easy to run. Truly, love it.
Re: R3, a map-reduce engine with Python and Redis
#28Anyone have some insight into situations where running map reduce on redis makes more sense than other software like the traditional hadoop?
If you have alot of data, and network IO is a big issue, you'll want to use something like hadoop (or disco) becuase they come with an integrated distributed file system and they preserve data locality.
If you don't have that much data, MR on redis is fine
Re: R3, a map-reduce engine with Python and Redis
#29Can multiple users run tasks simultaneously? Can they set task priorities?
Re: R3, a map-reduce engine with Python and Redis
#30Earlier quoted context omitted.
I'd love to, but it would take about an hour to run through everything. Here's a short version: There's a collective ecosystem problem of fragmented applications, not-quite-right command line utilities, web interfaces that look like they were designed in 1995, noisy log files people actually have to read constantly, and cross coupling of dependencies that make keeping a cluster live for production use a full time job…
seiji pretty much nails it. Hadoop seems to have come out of a weird culture. It is a distributed system with a single point of failure (name node) because its designers insisted on avoiding Paxos (distributed systems are too hard so we'll just make a broken-by-design protocol instead). Another example is that a lot of the database code built on top of Hadoop is designed around one Java hashmap per row which really l…
Wait, so Zookeeper (= distributed consensus thingie that I think implements the Paxos algorithm) is a Hadoop project but not actually used in Hadoop mapreduce?