Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hadoop / MapReduce alternatives for parallel computing?

tompinckney.com

Post reply on HN
Print view View original

1–10 of 12 posts

Re: Hadoop / MapReduce alternatives for parallel computing?

#3

Post by tom_pinckney »

nethergoat wrote: post #2

Deepak Singh of Amazon Web Services maintains a great list of (cloud-focused) parallel computing frameworks and platforms: http://deepaksingh.net/Resources/Computing_in_the_Cloud He's on twitter, too: http://twitter.com/mndoci

In the name of completeness, there are also great packages like OpenMPI and OpenMP.

At least for my particular applications, though, there's either 1) a steep learning curve for programmers 2) language support issues 3) they're designed for batch processing.

Personally, I find shared memory interfaces the easiest to program when there're complicated data access patterns. But that just might be personal preference.

Re: Hadoop / MapReduce alternatives for parallel computing?

#4

Post by amock »

tom_pinckney wrote: post #3
nethergoat wrote: post #2

Deepak Singh of Amazon Web Services maintains a great list of (cloud-focused) parallel computing frameworks and platforms: http://deepaksingh.net/Resources/Computing_in_the_Cloud He's on twitter, too: http://twitter.com/mndoci

In the name of completeness, there are also great packages like OpenMPI and OpenMP. At least for my particular applications, though, there's either 1) a steep learning curve for programmers 2) language support issues 3) they're designed for batch processing. Personally, I find shared memory interfaces the easiest to program when there're complicated data access patterns. But that just might be personal preference.

Shared memory interfaces are easy to use, but they don't support the same platform as Hadoop and MapReduce because you can't efficiently split them up across machines. With a distribute system like Hadoop you can build a cluster of cheap machines and spread the computation across them. With a shared memory architecture you have to scale up with multi-million dollar machines like SGI's altix line. So if you want to be able to use a cloud of cheap computers you need something like MPI or Hadoop.

Re: Hadoop / MapReduce alternatives for parallel computing?

#6

Post by tom_pinckney »

amock wrote: post #4
tom_pinckney wrote: post #3

Earlier quoted context omitted.

In the name of completeness, there are also great packages like OpenMPI and OpenMP. At least for my particular applications, though, there's either 1) a steep learning curve for programmers 2) language support issues 3) they're designed for batch processing. Personally, I find shared memory interfaces the easiest to program when there're complicated data access patterns. But that just might be personal preference.

Shared memory interfaces are easy to use, but they don't support the same platform as Hadoop and MapReduce because you can't efficiently split them up across machines. With a distribute system like Hadoop you can build a cluster of cheap machines and spread the computation across them. With a shared memory architecture you have to scale up with multi-million dollar machines like SGI's altix line. So if you want to be…

memcached is a poor-man's distributed shared memory system for clusters. We've been layering on top of it to try and fix deficiencies with things like client-side caching, persistence in case memcached drops objects etc.

But I was curious if other people had similar problems and how they were solving them.

Re: Hadoop / MapReduce alternatives for parallel computing?

#7

Post by antirez »

tom_pinckney wrote: post #6
amock wrote: post #4

Earlier quoted context omitted.

Shared memory interfaces are easy to use, but they don't support the same platform as Hadoop and MapReduce because you can't efficiently split them up across machines. With a distribute system like Hadoop you can build a cluster of cheap machines and spread the computation across them. With a shared memory architecture you have to scale up with multi-million dollar machines like SGI's altix line. So if you want to be…

memcached is a poor-man's distributed shared memory system for clusters. We've been layering on top of it to try and fix deficiencies with things like client-side caching, persistence in case memcached drops objects etc. But I was curious if other people had similar problems and how they were solving them.

Sounds like your life would be simpler with Redis if you are using memcached to take state about a computation. Atomic operations on lists and persistence are two good points about it in this context.

Re: Hadoop / MapReduce alternatives for parallel computing?

#8

Post by jerf »

tom_pinckney wrote: post #6
amock wrote: post #4

Earlier quoted context omitted.

Shared memory interfaces are easy to use, but they don't support the same platform as Hadoop and MapReduce because you can't efficiently split them up across machines. With a distribute system like Hadoop you can build a cluster of cheap machines and spread the computation across them. With a shared memory architecture you have to scale up with multi-million dollar machines like SGI's altix line. So if you want to be…

memcached is a poor-man's distributed shared memory system for clusters. We've been layering on top of it to try and fix deficiencies with things like client-side caching, persistence in case memcached drops objects etc. But I was curious if other people had similar problems and how they were solving them.

If you're trying to "fix" memcached's dropping of objects after a time, you shouldn't be using memcached. You should be using something actually designed to persist things. Turning a cache into a persistent store, or vice versa, is a dangerous game, and of the two cache -> persistent store is the worse.

Re: Hadoop / MapReduce alternatives for parallel computing?

#10

Post by hyuen »

tom_pinckney wrote: post #6
amock wrote: post #4

Earlier quoted context omitted.

Shared memory interfaces are easy to use, but they don't support the same platform as Hadoop and MapReduce because you can't efficiently split them up across machines. With a distribute system like Hadoop you can build a cluster of cheap machines and spread the computation across them. With a shared memory architecture you have to scale up with multi-million dollar machines like SGI's altix line. So if you want to be…

memcached is a poor-man's distributed shared memory system for clusters. We've been layering on top of it to try and fix deficiencies with things like client-side caching, persistence in case memcached drops objects etc. But I was curious if other people had similar problems and how they were solving them.

I have tried to use hadoop at lizten.in, but it seems overkill specially for small deployments like the basic slicehost option. I tried memcached and sure, it is a good fix for some page rendering issues, but for elements that you want to cache forever (or a relatively long time), I prefer to create a blob table in mysql and store the entries as if it was a key/value pair. I am sure there are more sophisticated persistent datastores, but I guess the idea is the same. Maybe with some high performance storage, such as SSD's and something like berkeleydb it would be feasible to keep a relatively big cache.
Post reply on HN