Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

R3, a map-reduce engine with Python and Redis

heynemann.github.com

Post reply on HN
Print view View original

31–40 of 40 posts

Re: R3, a map-reduce engine with Python and Redis

#31

Post by mumrah »

sqrt17 wrote: post #30
MichaelSalib wrote: post #12

Earlier quoted context omitted.

seiji pretty much nails it. Hadoop seems to have come out of a weird culture. It is a distributed system with a single point of failure (name node) because its designers insisted on avoiding Paxos (distributed systems are too hard so we'll just make a broken-by-design protocol instead). Another example is that a lot of the database code built on top of Hadoop is designed around one Java hashmap per row which really l…

Wait, so Zookeeper (= distributed consensus thingie that I think implements the Paxos algorithm) is a Hadoop project but not actually used in Hadoop mapreduce?

That's correct. I believe they are using it in some new "high availability" stuff coming down the road

Re: R3, a map-reduce engine with Python and Redis

#34

Post by jdboyd »

cypherpunks01 wrote: post #25

Do the map-reduce results get put back into redis? I always worry about OOM problems when I'm putting a somewhat unbounded set of things into redis.

You can configure redis to limit how much memory is used.

maxmemory 104857600

Of course, that still might not be the result you want.

Re: R3, a map-reduce engine with Python and Redis

#35

Post by heynemann »

fsaintjacques wrote: post #8
seiji wrote: post #4

Earlier quoted context omitted.

Hadoop is a bloated pile of elephant poo. Any and all alternatives are welcome. Disco ( http://discoproject.org/ ) is popular in some parts of the mapreducesphere.

Using disco here, very happy with it.

The reason I wrote r³ is because I was a little overwhelmed by how complex disco is to administer and scale.

r³ was designed from the ground up to adhere to HTTP. That means it's pretty easy to scale using our old and well-proven techniques: caching and load-balancing.

Re: R3, a map-reduce engine with Python and Redis

#37

Post by heynemann »

fsaintjacques wrote: post #10

Can you horizontaly scale the redis backend or it supports only one instance? Why restrain to sequential reducers when you can parallelize with partitions and sorting?

We do horizontally scale redis as a farm. I'll try to get more details on how we do it as I'm not the one responsible.

We thought of parallel reducers and it does make a lot of sense. The reason they are sequential is to get a first release out so we can juggle ideas with people. If you care to contribute we'd love it. Even if you just create an issue.

Re: R3, a map-reduce engine with Python and Redis

#38

Post by heynemann »

dchichkov wrote: post #29

Can multiple users run tasks simultaneously? Can they set task priorities?

Yes and No.

We use tornado for the stream (the task processor). That means that only one user gets to run a task simultaneously.

That said, the stream is just an http application.

This means that you can scale it as easily as you would any web app.

Re: R3, a map-reduce engine with Python and Redis

#39

Post by heynemann »

ericmoritz wrote: post #7
velodrome wrote: post #2

This looks like an interesting project. Is there something like this for php?

Python isn't a hard language to learn. It's probably easier to learn Python than to port this to PHP.

I agree, but one of the next features we'll implement is for you to be able to write stream processors, mappers and reducers in any language you want. Stay tuned!

Re: R3, a map-reduce engine with Python and Redis

#40

Post by heynemann »

jdboyd wrote: post #34
cypherpunks01 wrote: post #25

Do the map-reduce results get put back into redis? I always worry about OOM problems when I'm putting a somewhat unbounded set of things into redis.

You can configure redis to limit how much memory is used. maxmemory 104857600 Of course, that still might not be the result you want.

They do get put into redis.

Maybe we should have a different storage strategy if the data is too big? File storage? I just meant for it to be simple.

If you are going to use redis for storage then you'll need to fine tune it to the processing you are doing (we have).

Post reply on HN