Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: To everybody who uses MapReduce: what problems do you solve?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 122 posts

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#81

Post by ameen »

lclarkmichalek wrote: post #28

This system isn't in production just yet, but should be shortly. We're parsing Dota2 replays and generating statistics and visualisation data from them, which can then be used by casters and analysts for tournaments, and players. The replay file format breaks the game down into 1 min chunks, which are the natural thing to iterate over. Before someone comes along and says "this isn't big data!", I know. It's medium da…

Are you talking about match stastics or player stats? Dota 2 has a logging system that records various events? during a match.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#82

Post by alecco »

AsymetricCom wrote: post #78
alecco wrote: post #77

Earlier quoted context omitted.

Is this serious? Have you ported a program to Hadoop? Unles you use Pig or one of those helping layers it is quite hard for non-trivial problems. And those helping layers usually come with some overhead cost for non-trivial cases, too. Edit: no downvote from me.

It was a pretty easy problem, parsing logs for performance statistics. But moving the data is the easy part and that's why I was incredulous of the OP's statement. I'm starting to wonder if this is really "Hacker News" or if it's "we want free advice and comments from engineers on our startups so lets start a forum with technical articles"

Big Data should be on the Peta+ level. Even with 10G Ethernet it takes a lot of bandwidth and time to move things around (and it's very hard to keep 10G ethernet full at a constant rate from storage). This is hard even for telcos. Note Terabyte+ level today fits on SSD.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#84

Post by alecco »

aidos wrote: post #74
alecco wrote: post #55

Earlier quoted context omitted.

Maybe I misunderstood what you were asking. Note both MapReduce and NoSQL are overhyped solutions. They are useful in a handful of cases, but often applied to problems they are not as good.

I'm not sure that the two concepts are resulted at all. Obviously mongo has map reduce baked in - but that's not that relevant. Map/reduce is a reasonable paradigm for crunching information. I have a heavily CPU bound system that I parallelise by running on different machines and aggregating that results. I probably wouldn't call it map reduce - but really it's the same thing. How do you parallelise your long running…

I can't say without more information on the problem to solve. As I said above, there are cases where MapReduce is a good tool.

And even if you improve the DS/Algorithm first, usually that is usable by the MapReduce port and you save a lot of time/costs.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#85

Post by msellout »

collyw wrote: post #26
thrownaway2424 wrote: post #19

Earlier quoted context omitted.

I strongly agree. Although there are clearly uses for map/reduce at large scale there is also a tendency to use it for small problems where the overhead is objectionable. At work I've taken multiple Mao/reduce systems and converted them to run on my desktop, in one case taking a job that used to take 5 minutes just to startup down to a few seconds total. Right tool for the job and all that. If you need to process a 5…

I completely agree as well, but I don't consider myself much of an expert in NoSQL technologies (which is why I read up on threads like this to find out). Does anyone have a use case where data is on a single machine and map reduce is still relevant? (I am involved in a project at work where the other guys seem to have enthusiastically jumped on MongoDB without great reasons in my opinion).

You can imagine cases where map-reduce is useful without any starting data. If you are analyzing combinations or permutations, you can create a massive amount of data in an intermediate step, even if the initial and final data sets are small.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#86

Post by alexatkeplar »

sandGorgon wrote: post #76
alexatkeplar wrote: post #67

We use Elastic MapReduce at Snowplow to validate and enrich raw user events (collected in S3 from web, Lua, Arduino etc clients) into "full fat" schema'ed Snowplow events containing geo-IP information, referer attribution etc. We then load those events from S3 into Redshift and Postgres. So we are solving the problem of processing raw user behavioural data at scale using MapReduce. All of our MapReduce code is writte…

thanks for this. snowplow has been an amazing source of learning. I'm quite interested in the etl process than in the actual mapreduce. have you seen your etl used to pull data from Twitter or Facebook. I am wondering what is the state of art there considering throttling, etc.

Hi sandGorgon! Thanks for the encouraging words. We haven't yet seen people use the existing Scalding ETL to pull data from Twitter or Facebook. As you suggest, there are some considerations around using Hadoop to access web APIs without getting throttled/banned. Here's a couple of links which might be helpful:

- http://stackoverflow.com/questions/6206105/running-web-fetch...

- http://petewarden.com/2011/05/02/using-hadoop-with-external-...

I think a gatekeeper service could make sense; or alternatively you could write something which runs prior to your MapReduce job and e.g. just loads your results into HDFS/HBase, for the MapReduce to then lookup into. Akka or maybe Storm could be choices here.

We have done a small prototype project to pull data out of Twitter & Facebook - that was only a Python/RDS pilot, but it gave us some ideas for how a proper social feed into Snowplow could work.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#87

Post by _random_ »

kamakazizuru wrote: post #45
_random_ wrote: post #40

Earlier quoted context omitted.

Hearing to what problems a technology failed to solve is usually even more interesting. It cuts the hype.

sure - but that's not the original question. Maybe the OP is just curious to understand some in production use cases (and the approach to their implementation) - and isn't that interested to know the edge cases where it doesn't work.

Maybe his case is an edge case.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#88

Post by prawks »

ameen wrote: post #81
lclarkmichalek wrote: post #28

This system isn't in production just yet, but should be shortly. We're parsing Dota2 replays and generating statistics and visualisation data from them, which can then be used by casters and analysts for tournaments, and players. The replay file format breaks the game down into 1 min chunks, which are the natural thing to iterate over. Before someone comes along and says "this isn't big data!", I know. It's medium da…

Are you talking about match stastics or player stats? Dota 2 has a logging system that records various events? during a match.

Replays contain more detailed data (I believe) like player positioning and actions. I think logging is pretty limited.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#89

Post by lclarkmichalek »

ameen wrote: post #81
lclarkmichalek wrote: post #28

This system isn't in production just yet, but should be shortly. We're parsing Dota2 replays and generating statistics and visualisation data from them, which can then be used by casters and analysts for tournaments, and players. The replay file format breaks the game down into 1 min chunks, which are the natural thing to iterate over. Before someone comes along and says "this isn't big data!", I know. It's medium da…

Are you talking about match stastics or player stats? Dota 2 has a logging system that records various events? during a match.

Match stats atm, though we can get much everything. We can go far beyond the combat log, reproducing an entire minimap, along with things like hero position trails, map control, etc etc. Here's a screenshot of a minimap recreation we produced for the recent Fragbite Masters tournament: http://i.imgur.com/kuOvUZX.jpg. We have more complex tools such as XP and Gold source breakdowns over time (i.e. jungle creeps, lane creeps, kills, assists etc), along with other fun stuff still in the pipeline.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#90

Post by msellout »

collyw wrote: post #57
baudehlo wrote: post #44

Earlier quoted context omitted.

This sounds like something you could just do in SQL and have it all done in milliseconds.

Most of the NoSQL cases I have heard seem to be that they could be done at least as well in SQL. I have asked a lot of questions on this topic, and no one has yet convinced me (please do if you have a legitimate NoSQL case).

Execution time, after a point, is less important than development time. NoSQL is often faster for development and refactoring, because the schema is easier to change in code than in the database.

Also there's the scale issue. Append-only is helpful. Why have SQL if you can't use all the features?

Post reply on HN