Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

R3, a map-reduce engine with Python and Redis

heynemann.github.com

Post reply on HN
Print view View original

11–20 of 40 posts

Re: R3, a map-reduce engine with Python and Redis

#11

Post by cgh »

seiji wrote: post #9
achompas wrote: post #6

Earlier quoted context omitted.

I hear the above comment about Hadoop a lot. Can you explain why?

I'd love to, but it would take about an hour to run through everything. Here's a short version: There's a collective ecosystem problem of fragmented applications, not-quite-right command line utilities, web interfaces that look like they were designed in 1995, noisy log files people actually have to read constantly, and cross coupling of dependencies that make keeping a cluster live for production use a full time job…

I'd also just like to say: NameNode = single point of failure.

I worked on a contract for a large, very well-known social networking company a while back who refused to consider Hadoop because of this.

Re: R3, a map-reduce engine with Python and Redis

#12

Post by MichaelSalib »

seiji wrote: post #9
achompas wrote: post #6

Earlier quoted context omitted.

I hear the above comment about Hadoop a lot. Can you explain why?

I'd love to, but it would take about an hour to run through everything. Here's a short version: There's a collective ecosystem problem of fragmented applications, not-quite-right command line utilities, web interfaces that look like they were designed in 1995, noisy log files people actually have to read constantly, and cross coupling of dependencies that make keeping a cluster live for production use a full time job…

seiji pretty much nails it. Hadoop seems to have come out of a weird culture. It is a distributed system with a single point of failure (name node) because its designers insisted on avoiding Paxos (distributed systems are too hard so we'll just make a broken-by-design protocol instead). Another example is that a lot of the database code built on top of Hadoop is designed around one Java hashmap per row which really limits performance.

There are all sorts of oddities and you can mostly work around them but it is...exhausting, and I spend a lot of time thinking "surely there must be a better way".

Re: R3, a map-reduce engine with Python and Redis

#14

Post by grantjgordon »

seiji wrote: post #9
achompas wrote: post #6

Earlier quoted context omitted.

I hear the above comment about Hadoop a lot. Can you explain why?

I'd love to, but it would take about an hour to run through everything. Here's a short version: There's a collective ecosystem problem of fragmented applications, not-quite-right command line utilities, web interfaces that look like they were designed in 1995, noisy log files people actually have to read constantly, and cross coupling of dependencies that make keeping a cluster live for production use a full time job…

Thanks for your insightful comments! I appreciate that you took the time to back up your opinion by distilling your thoughts into something quickly digestible.

Have you heard of any other projects outside of disco that are more performant than hadoop when used for similar applications?

Re: R3, a map-reduce engine with Python and Redis

#15

Post by grantjgordon »

fsaintjacques wrote: post #8
seiji wrote: post #4

Earlier quoted context omitted.

Hadoop is a bloated pile of elephant poo. Any and all alternatives are welcome. Disco ( http://discoproject.org/ ) is popular in some parts of the mapreducesphere.

Using disco here, very happy with it.

Mind sharing how long you've been using it and how it compares to hadoop in your opinion? I'm very interesting in hearing your experience.
Post reply on HN