Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: To everybody who uses MapReduce: what problems do you solve?

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 122 posts

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#111

Post by nl »

While the push-back against Map/Reduce & "Big Data" in general is totally valid, it's important to put it into context.

In 2007-2010, when Hadoop first started to gain momentum it was very useful because disk sizes were smaller, 64 bit machines weren't ubiquitous and (perhaps most importantly) SSDs were insanely expensive for anything more than tiny amounts of data.

That meant if you had more than a couple of terabytes of data you either invested in a SAN, or you started looking at ways to split your data across multiple machines.

HDFS grew out of those constraints, and once you have data distributed like that, with each machine having a decently powerful CPU as well, Map/Reduce is a sensible way of dealing with it.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#112

Post by smoyer »

DenisM wrote: post #110
smoyer wrote: post #101

Earlier quoted context omitted.

I've been using CouchDB (and now BigCouch) for about four years and it's both clever and useful. We're storing engineering documents (as attachments) and using map/reduce (CouchDB views) to segment the documents by the metadata stored in the fields. The only downside is that adding a view with trillions of rows can take quite a while.

Strangely, there aren't many discussions of Couch* family on hacker news. Do you know why would that be? I'm thinking about basing a new product around couchbase lite, but lack of popular acceptance is one of the things holding me back.

I'd be happy to take the discussion off-line if you want to get into more depth but I have to wonder if the merger of Memcached and CouchDB happened right as CouchDB would have made a name for itself. Right around that time, I think CouchDB and Mongo were equally well-known.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#113

Post by sandGorgon »

alexatkeplar wrote: post #86
sandGorgon wrote: post #76

Earlier quoted context omitted.

thanks for this. snowplow has been an amazing source of learning. I'm quite interested in the etl process than in the actual mapreduce. have you seen your etl used to pull data from Twitter or Facebook. I am wondering what is the state of art there considering throttling, etc.

Hi sandGorgon! Thanks for the encouraging words. We haven't yet seen people use the existing Scalding ETL to pull data from Twitter or Facebook. As you suggest, there are some considerations around using Hadoop to access web APIs without getting throttled/banned. Here's a couple of links which might be helpful: - http://stackoverflow.com/questions/6206105/running-web-fetch... - http://petewarden.com/2011/05/02/using-…

is your python code part of the snowplow repository/gist?

it would be very interesting to take a look at it.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#115

Post by petar »

alecco wrote: post #15

A large telco has a 600 node cluster of powerful hardware. They barely use it. Moving Big Data around is hard. Managing is harder. A lot of people fail to understand the overheads and limitations of this kind of architecture. Or how hard it is to program, especially considering salaries for this skyrocketed. More often than not a couple of large 1TB SSD PCIe and a lot of RAM can handle your "big" data problem. Before…

Consider using the gocircuit.org

It was expressly designed to avoid non-essential technical problems that come in the way of cloud application developers, when they try to orchestrate algorithms across multiple machines.

Identical HBASE SQL queries and their respective counterparts implemented in the Circuit Language are: Approximately equally long in code, and orders of magnitude faster than HBASE. In both cases, the data is pulled out of the same HBASE cluster for fair comparison.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#117

Post by continuations »

petar wrote: post #115
alecco wrote: post #15

A large telco has a 600 node cluster of powerful hardware. They barely use it. Moving Big Data around is hard. Managing is harder. A lot of people fail to understand the overheads and limitations of this kind of architecture. Or how hard it is to program, especially considering salaries for this skyrocketed. More often than not a couple of large 1TB SSD PCIe and a lot of RAM can handle your "big" data problem. Before…

Consider using the gocircuit.org It was expressly designed to avoid non-essential technical problems that come in the way of cloud application developers, when they try to orchestrate algorithms across multiple machines. Identical HBASE SQL queries and their respective counterparts implemented in the Circuit Language are: Approximately equally long in code, and orders of magnitude faster than HBASE. In both cases, th…

What does Go Circuit do differently that makes it orders of magnitude faster than HBase SQL?

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#118

Post by mfeldman »

"Things that could be trivially done in SQL :(" We use HIVE over HDFS. Sure the type of things we are doing could have been done in SQL, if we had carefully curated our data set and chosen which parts to keep and which to throw away. Unfortunately, we are greedy in the storage phase. Hive allows us to save much more than we reasonably would need, which actually is great when we need to get it long after the insert was made.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#119

Post by jfxberns »

alecco wrote: post #15

A large telco has a 600 node cluster of powerful hardware. They barely use it. Moving Big Data around is hard. Managing is harder. A lot of people fail to understand the overheads and limitations of this kind of architecture. Or how hard it is to program, especially considering salaries for this skyrocketed. More often than not a couple of large 1TB SSD PCIe and a lot of RAM can handle your "big" data problem. Before…

"A lot of people fail to understand the overheads and limitations of this kind of architecture. Or how hard it is to program, especially considering salaries for this skyrocketed. More often than not a couple of large 1TB SSD PCIe and a lot of RAM can handle your "big" data problem."

It's not that hard to program... it does take a shift in how you attack problems.

If your data set fits on a few SSDs. then you probably don't have a real big data problem.

"Moving Big Data around is hard. Managing is harder."

Moving big data around is hard--that's why you have hadoop--you send the compute to where he data is, thus requiring a new way of thinking about how you do computations.

"Before doing any Map/Reduce (or equivalent), please I beg you to check out Introduction to Data Science at Coursera https://www.coursera.org/course/datasci"

Data science does not solve the big data problem. Here's my favorite definition of a big data problem: "a big data problem is when the size of the data becomes part of the problem." You can't use traditional linear programming models to handle a true big data problem; you have to have some strategy to parallelize the compute. Hadoop is great for that.

"A large telco has a 600 node cluster of powerful hardware. They barely use it."

Sounds more like organizational issues, poo planning and execution than a criticism of Hadoop!

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#120

Post by jfxberns »

collyw wrote: post #26
thrownaway2424 wrote: post #19

Earlier quoted context omitted.

I strongly agree. Although there are clearly uses for map/reduce at large scale there is also a tendency to use it for small problems where the overhead is objectionable. At work I've taken multiple Mao/reduce systems and converted them to run on my desktop, in one case taking a job that used to take 5 minutes just to startup down to a few seconds total. Right tool for the job and all that. If you need to process a 5…

I completely agree as well, but I don't consider myself much of an expert in NoSQL technologies (which is why I read up on threads like this to find out). Does anyone have a use case where data is on a single machine and map reduce is still relevant? (I am involved in a project at work where the other guys seem to have enthusiastically jumped on MongoDB without great reasons in my opinion).

"Does anyone have a use case where data is on a single machine and map reduce is still relevant?"

No! MapReduce is a programming pattern for massively parallelizing computational tasks. If you are doing it on one machine, you are not massively parallelizing your compute and you don't need MapReduce.

Post reply on HN