Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: To everybody who uses MapReduce: what problems do you solve?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 122 posts

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#41

Post by Roboprog »

collyw wrote: post #34
alecco wrote: post #30

Earlier quoted context omitted.

> Does anyone have a use case where data is on a single machine and map reduce is still relevant? What matters is the running data structure. For example, you can have Petabytes of logs but you need a map/table of some kind to do aggregations/transformations. Or a sparse-matrix based model. There are types of problems that can partition the data structure and work in parallel in the RAM of many servers. Related: it's…

OK, maybe I am not understanding you correctly, but what you describe seems to be, if the data is on one machine, connect to a cluster of machines, and run processing in parallel on that. That doesn't imply a NoSQL solution to me. Just parallel processing on different parts of the data. If I am wrong can you point me to a clearer example?

It sounds to me like the poster above restructured the input data to exploit locality of reference better.

http://en.wikipedia.org/wiki/Locality_of_reference

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#42

Post by collyw »

res0nat0r wrote: post #20
srl wrote: post #16

Earlier quoted context omitted.

This doesn't answer the question - you described /how/ you solved some problem, not what problem you're actually solving. (Mind you, at my writing this is the top comment, so I don't think you're getting many downvotes. But your comment irked me, so there you go.)

> not what problem you're actually solving. > targetting expressions over billions of time series events from users.

You moved back to MySQL.

What was the motivation to move to Map Reduce in the first place if a well understood technology, MySQL, works fine?

(sorry if I am posting a lot on this topic. I am really interested in finding answers rather than trying to prove any point that relational databases are better in case anyone thinks otherwise).

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#43

Post by oacgnol »

zengr wrote: post #27

We use MR using Pig (data in cassandra/CFS) with a 6 node hadoop cluster to process timeseries data. The events contain user metrics like which view was tapped, user behavior, search result, clicks etc. We process these events to use it downstream for our search relevancy, internal metrics, see top products. We did this on mysql for a long time but things went really slow. We could have optimized mysql for performanc…

Can you estimate how much faster your processing is now vs. before on MySQL? I find it interesting that your cluster is only 6 nodes - relatively small compared to what I've seen and from what I've read about. It'd be interesting to know the benefits of small-scale usage of big data tech.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#44

Post by baudehlo »

lclarkmichalek wrote: post #28

This system isn't in production just yet, but should be shortly. We're parsing Dota2 replays and generating statistics and visualisation data from them, which can then be used by casters and analysts for tournaments, and players. The replay file format breaks the game down into 1 min chunks, which are the natural thing to iterate over. Before someone comes along and says "this isn't big data!", I know. It's medium da…

This sounds like something you could just do in SQL and have it all done in milliseconds.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#45

Post by kamakazizuru »

_random_ wrote: post #40
srl wrote: post #16

Earlier quoted context omitted.

This doesn't answer the question - you described /how/ you solved some problem, not what problem you're actually solving. (Mind you, at my writing this is the top comment, so I don't think you're getting many downvotes. But your comment irked me, so there you go.)

Hearing to what problems a technology failed to solve is usually even more interesting. It cuts the hype.

sure - but that's not the original question. Maybe the OP is just curious to understand some in production use cases (and the approach to their implementation) - and isn't that interested to know the edge cases where it doesn't work.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#46

Post by rubyfan »

We are using Map/Reduce to analyze raw XML as well as event activity streams, for example analyzing a collection of events and meta data to understand how discreet events relate to each other as well as patterns leading to certain outcomes. I am primarily using Ruby+Wukong via the Hadoop-Streaming interface as well as Hive to analyze output and for more normalized data problems.

The company is a large Fortune 500 P&C insurer and has a small (30 node) Cloudera 4 based cluster in heavy use by different R&D, analytic and technology groups within the company. Those other groups use a variety of toolsets in the environment, I know of Python, R, Java, Pig, Hive, Ruby in use as well as more traditional tools on the periphery in the BI and R&D spaces such as Microstrategy, Ab Initio, SAS, etc.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#48

Post by collyw »

rch wrote: post #39
collyw wrote: post #36

Earlier quoted context omitted.

So is it fair to say it is an "ease of use" use case?

Is that the same as an 'impossible to do otherwise' case? Edit: I should say 'currently impossible' since as I noted, I can imagine being able to build SQL queries around PSSM comparisons and the like. I just can't build a system to last 5+ years around something that might be available at some point. Since I can't reply directly- agreed :)

That comment was based on your "into Postgres wouldn't be impossible" phrase.

No fair enough of its the only way you can get things to work. I still see lots of people jumping on the "big data" bandwagon with very moderate sized data.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#49

Post by Roboprog »

markuskobler wrote: post #32
collyw wrote: post #26

Earlier quoted context omitted.

I completely agree as well, but I don't consider myself much of an expert in NoSQL technologies (which is why I read up on threads like this to find out). Does anyone have a use case where data is on a single machine and map reduce is still relevant? (I am involved in a project at work where the other guys seem to have enthusiastically jumped on MongoDB without great reasons in my opinion).

On modern hardware with many cpu cores you can use a similar process of fork and join to maximise throughput of large datasets.

Forks/threads on (e.g.) 12 core CPUs works up to a point. But that point probably does solve many problems without further complication :-)

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#50

Post by shamsulbuddy »

joeblau wrote: post #17

I was using it on a D3.JS chart to aggregate data flow though our custom real-time analytic pipeline.

Will u mind sharing some more info on this real time analytics using d3 as i am also planning to do something like tht..
Post reply on HN