Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: To everybody who uses MapReduce: what problems do you solve?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 122 posts

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#51

Post by willvarfar »

collyw wrote: post #42
res0nat0r wrote: post #20

Earlier quoted context omitted.

> not what problem you're actually solving. > targetting expressions over billions of time series events from users.

You moved back to MySQL. What was the motivation to move to Map Reduce in the first place if a well understood technology, MySQL, works fine? (sorry if I am posting a lot on this topic. I am really interested in finding answers rather than trying to prove any point that relational databases are better in case anyone thinks otherwise).

(Poster) we moved from mysql+innodb/myisam to hadoop because of performance problems. We did test and evaluate hadoop, and then jumped. Then tokudb comes along (technically we moved back to mariadb) and puts performance advantage firmly back on mysql's side. I imagine impala and presto and any other column based, non-map-reduce engines would give tokudb a fairer fight though.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#52

Post by alecco »

Roboprog wrote: post #41
collyw wrote: post #34

Earlier quoted context omitted.

OK, maybe I am not understanding you correctly, but what you describe seems to be, if the data is on one machine, connect to a cluster of machines, and run processing in parallel on that. That doesn't imply a NoSQL solution to me. Just parallel processing on different parts of the data. If I am wrong can you point me to a clearer example?

It sounds to me like the poster above restructured the input data to exploit locality of reference better. http://en.wikipedia.org/wiki/Locality_of_reference

It's one of the issues, yes. But I wanted to be more general on purpose.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#53

Post by Aqueous »

To most people who use MapReduce in a cluster: You probably don't need to use MapReduce. You are either vastly overstating the amount of data you are dealing with and the complexity of what you need to do with that data, or you are vastly understating the amount of computational power a single node actually has. Either way, see how fast you can do it on a single machine before trying to run it on a cluster.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#54

Post by collyw »

Roboprog wrote: post #41
collyw wrote: post #34

Earlier quoted context omitted.

OK, maybe I am not understanding you correctly, but what you describe seems to be, if the data is on one machine, connect to a cluster of machines, and run processing in parallel on that. That doesn't imply a NoSQL solution to me. Just parallel processing on different parts of the data. If I am wrong can you point me to a clearer example?

It sounds to me like the poster above restructured the input data to exploit locality of reference better. http://en.wikipedia.org/wiki/Locality_of_reference

So assuming the data is one one machine (as I asked), why would an index not solve this problem? And why does Map Reduce solve it?

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#55

Post by alecco »

collyw wrote: post #34
alecco wrote: post #30

Earlier quoted context omitted.

> Does anyone have a use case where data is on a single machine and map reduce is still relevant? What matters is the running data structure. For example, you can have Petabytes of logs but you need a map/table of some kind to do aggregations/transformations. Or a sparse-matrix based model. There are types of problems that can partition the data structure and work in parallel in the RAM of many servers. Related: it's…

OK, maybe I am not understanding you correctly, but what you describe seems to be, if the data is on one machine, connect to a cluster of machines, and run processing in parallel on that. That doesn't imply a NoSQL solution to me. Just parallel processing on different parts of the data. If I am wrong can you point me to a clearer example?

Maybe I misunderstood what you were asking.

Note both MapReduce and NoSQL are overhyped solutions. They are useful in a handful of cases, but often applied to problems they are not as good.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#56

Post by alecco »

zengr wrote: post #27

We use MR using Pig (data in cassandra/CFS) with a 6 node hadoop cluster to process timeseries data. The events contain user metrics like which view was tapped, user behavior, search result, clicks etc. We process these events to use it downstream for our search relevancy, internal metrics, see top products. We did this on mysql for a long time but things went really slow. We could have optimized mysql for performanc…

Did you try a columnar database first? Vertica, InfinyDB, MonetDB... There are many.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#57

Post by collyw »

baudehlo wrote: post #44
lclarkmichalek wrote: post #28

This system isn't in production just yet, but should be shortly. We're parsing Dota2 replays and generating statistics and visualisation data from them, which can then be used by casters and analysts for tournaments, and players. The replay file format breaks the game down into 1 min chunks, which are the natural thing to iterate over. Before someone comes along and says "this isn't big data!", I know. It's medium da…

This sounds like something you could just do in SQL and have it all done in milliseconds.

Most of the NoSQL cases I have heard seem to be that they could be done at least as well in SQL.

I have asked a lot of questions on this topic, and no one has yet convinced me (please do if you have a legitimate NoSQL case).

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#58

Post by lclarkmichalek »

baudehlo wrote: post #44
lclarkmichalek wrote: post #28

This system isn't in production just yet, but should be shortly. We're parsing Dota2 replays and generating statistics and visualisation data from them, which can then be used by casters and analysts for tournaments, and players. The replay file format breaks the game down into 1 min chunks, which are the natural thing to iterate over. Before someone comes along and says "this isn't big data!", I know. It's medium da…

This sounds like something you could just do in SQL and have it all done in milliseconds.

For our highly unstructured, untyped, non relational artefacts? It'd be fairly impossible to use it as a datastore in this particular case, and regardless, it would provide little to no speed increase over our current application, as the limiting factor is the CPU cost of the map function.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#60

Post by DenisM »

It's worth noting that CouchDB is using map-reduce to define materialized views. Whereas normally MR parallelization is used to scale out, in this case it's used instead to allow incremental updates for the materialized views, which is to say incremental updates for arbitrarily defined indexes! By contrast SQL databases allow incremental updates only for indexes whose definition is well understood by the database engine. I found this to be pretty clever.
Post reply on HN